Flow Matching for Generative Modeling
Authors: Yaron Lipman
Ricky T. Q. Chen
Heli Ben-Hamu
Maximilian Nickel
Matthew Le
Venue: ICLR 2023 notable top 25% (2023)
Keywords: continuous normalizing flows
generative models

Abstract: We introduce a new paradigm for generative modeling built on Continuous Normalizing Flows (CNFs), allowing us to train CNFs at unprecedented scale. Specifically, we present the notion of Flow Matching (FM), a simulation-free approach for training CNFs based on regressing vector fields of fixed conditional probability paths. Flow Matching is compatible with a general family of Gaussian probability paths for transforming between noise and data samples---which subsumes existing diffusion paths as specific instances. Interestingly, we find that employing FM with diffusion paths results in a more robust and stable alternative for training diffusion models. Furthermore, Flow Matching opens the door to training CNFs with other, non-diffusion probability paths. An instance of particular interest is using Optimal Transport (OT) displacement interpolation to define the conditional probability paths. These paths are more efficient than diffusion paths, provide faster training and sampling, and result in better generalization. Training CNFs using Flow Matching on ImageNet leads to consistently better performance than alternative diffusion-based methods in terms of both likelihood and sample quality, and allows fast and reliable sample generation using off-the-shelf numerical ODE solvers.

Published as a conference paper at ICLR 2023
FLOW MATCHING FOR GENERATIVE MODELING
Yaron Lipman 1.2 Ricky T. Q. Chen' Heli Ben-Hamu? Maximilian Nickel Matt Le
Meta AI (FAIR) 2Weizmann Institute of Science
ABSTRACT
We introduce new paradigm for generative modeling built on Continuous Normalizing Flows (CNFs), allowing us to train CNFs at unprecedented scale. Specificallyy, we present the notion of Flow Matching (FM), a simulation-frre approach for training CNFs based on regressing vector fields of fixed conditional probability paths. Flow Matching is compatible with general family of Gaussian probability paths for transforming between noise and data samples-which -whic subsumes existing diffusion paths specific instances. Interestingly, that employing FM with diffusion paths results in a more robust and stable alternative for training diffusion models. Furthermore, Flow Matching opens the door to training CNFs with other, non-diffusion probability paths. An instance of particular interest using Optimal Transport (OT) displacement interpolation to define the conditional probability paths. These paths are more efficient than diffusion paths, provide faster training and sampling, and result better generalization. Training CNFs using Flow Matching ImageNet leads to consistently better performance than alternative diffusion- methods terms of both likelihood and sample quality, and allows fast and reliable sample generation using off- the-shelf numerical ODE solvers.
find
INTRODUCTION
Deep generative models are class of deep learning algorithms aimed estimating and sampling from an unknown data distribution. The recent influx of amazing advances in generative modeling, e.g. for image generation Ramesh al. (2022); Rombach al. (2022), is mostly facilitated by the scalable and relatively stable training of diffusion-based models Ho et al. (2020); Song et al. (2020b). However, the restriction to simple diffusion processes leads rather confined space of sampling probability paths, resulting in very long training times and the need to adopt specialized methods (e.g., Song et al. (2020a); Zhang & Chen (2022)) for efficient sampling.
In this work we consider the general and deterministic framework of Continuous Normalizing Flows (CNFs; Chen al. (2018)). CNF' are capable of modeling arbitrary probability path and are in par- ticular known to encompass the probability paths modeled by diffusion processes (Song et al., 202 However, aside from diffusion that can be trained efficiently via, e-g-, denoising score matching (Vin- cent, 2011), no scalable CNF training algorithms are known. Indeed, maximum likelihood train- ing (e.g., Grathwohl et al. (2018)) require expen- sive numerical ODE simulations, while existing simulation-free methods either involve intractable integrals (Rozen al., 2021) or biased gradients (Ben-Hamu et al., 2022).
The goal of this work is to propose Flow Matching (FM), an efficient simulation-free approach to train- ing CNF models, allowing the adoption of general probability paths to supervise CNF training. Im- portantly, FM breaks the barriers for scalable CNF training beyond diffusion, and sidesteps the need to reason about diffusion processes to directly work with probability paths.
Figure 1: Unconditional ImageNet-128 sam- ples of a CNF trained using Flow Matching with Optimal Transport probability paths.

---

Published as a conference paper at ICLR 2023
In particular, we propose the Flow Matching objective (Section 3), simple and intuitive training objective to regress onto a target vector field that generates a desired probability path. We first show that can construct such target vector fields through per-example (i.e,, conditional) formu- lations. Then, inspired by denoising score matching, we show that a per-example training objective, termed Conditional Flow Matching (CFM), provides equivalent gradients and does not require plicit knowledge of the intractable target vector field. Furthermore, we discuss a general family of per-example probability paths (Section 4) that can be used for Flow Matching, which subsumes ex- isting diffusion paths as special instances. Even on diffusion paths, we find that using FM provides more robust and stable training, and achieves superior performance compared to score matching. Furthermore, this family of probability paths also includes a particularly interesting case: the vector field that corresponds to an Optimal Transport (OT) displacement interpolant (McCann, 1997). We find that conditional OT paths are simpler than diffusion paths, forming straight line trajectories whereas diffusion paths result in curved paths. These properties seem to empirically translate to faster training, faster generation, and better performance.
We empirically validate Flow Matching and the construction via Optimal Transport paths on Im- ageNet, large and highly diverse image dataset. We find that we easily train models to achieve favorable performance in both likelihood estimation and sample quality amongst competing ceffusion-based methods. Furthermore, computational cost and sample quality compared to prior methods. Figure depicts selected uncon- ditional ImageNet 128x 128 samples from our model.
find that our models produce better trade-offs between
PRELIMINARIES: CONTINUOUS NORMALIZING FLOWS
Let Rd denote the data space with data points x = ( I Rd. Two important objects we use in this paper are: the probability density path p [0 x Rd + Ryo, which is a time dependent' probability density function, i.e., (e(x)) = 1, and time-dependent vector field, [0, x Rd Rd A vector field 2 be used to construct dime-dependent diffeomorphic map, called a flow, d= [0, 1] x Rd + Rdd defined via the ordinary differential equation (ODE):
.....
d
dt 0x(x)) = (.(((z)))
6o(z)= =
(2)
Previously, Chen et al. (2018) suggested modeling the vector field v with neural network, (z 0), where 0 E IRP are its learnable parameters, which in turn leads to a deep parametric model of the flow di. called Continuous Normalizing Flow (CNF). A CNF is used to reshape simple prior density Po (e.g., pure noise) to a more complicated one, P1, via the push-forward equation
Pt = [0t]-Po
(3)
where the forward (or change of variables) operator is defined by
[o)).Po(z) = po (x:) det T (x))
(4)
A vector field is is said to generate probability density path pa if its flow di satisfies equation 3. One practical way to test if a vector field generates probability path is using the continuity equation, which is a key component in our proofs, see Appendix A. We recap more information on CNFs, in particular how to compute the probability Pı (Ix)) at an arbitrary point v E Rd in Appendix C.
3 FLOW MATCHING
Let I1 denote a random variable distributed according to some unknown data distribution g(11). We assume we only have access to data samples from g(²1) but have access the density function itself. Furthermore, we let Pt be a probability path such that po = p is a simple distribution, egg., the standard normal distribution p(x:) N (==00, T), and let Pı be approximately equal in distribution to We will later discuss how to construct such path. The Flow Matching objective is then designed match this target probability path, which will allow us flow from po to P1.
We use subscript to denote the time parameter, e.g., (x))
2

---

Published as a conference paper at ICLR 2023
Given target probability density path pa (x) and corresponding vector field uz(z), which generates Pe(z), we define the Flow Matching (FM) objective as
L (0) p))))( (x)) us (x)]1²
(5)
where 9 denotes the learnable parameters of the CNF vector field et (as defined in Section 2), 21[0, 1] (uniform distribution), and x Pr (3)). Simply put, the FM loss regresses the vector field We with a neural network Ut. Upon reaching zero loss, the learned CNF model will generate pt (x).
Flow Matching is simple and attractive objective, but naïvely on its own, it is intractable to use in practice since we have no prior knowledge for what an appropriate Pr and ", are. There are many generally
choices of probability paths that can satisfy Pı (z) z q(x), and more importantly, don't have access to a closed form u that generates the desired Pt- In this section, we show that construct both pt and de using probability paths and vector fields that are only defined per sample, and an appropriate method of aggregation provides the desired p. and ug. Furthermore, this construction allows to create much more tractable objective for Flow Matching.
3.1 CONSTRUCTING Pt, FROM CONDITIONAL PROBABILITY PATHS AND VECTOR FIELDS
A simple way to construct a target probability path is via a mixture of simpler probability paths: Given particular data sample In denote by Pe (2|221 conditional probability path such that satisfies Po (2/221 = p(x) at time t = O, and we design p1 (x|(1) at f to be a distribution concentrated around = 3i, e.g., pı (
a normal distribution with æ1 mean and a sufficiently small standard deviation > O. Marginalizing the conditional probability paths over ((x1) give rise to the marginal probability path
px(x) - =
(6)
where in particular at time 1, the marginal probability pi mixture distribution that closely approximates the data distribution g.
(T f (1(1(11)((1)ddr)) g(z).
Interestingly, we can also define a marginal vector field, by "marginalizing" over the conditional vector fields in the following sense (we assume pe(x) > 0 for all t and æ):
-mn(attar) (M((a(aa)) Pt Pe(30lar p((3)(11)((111) dri,
ue (x) =
pe(x)
where uu (-(22) Rdd Rdd is a conditional vector field that generates pr(-(x1). It may not seem apparent, but this way of aggregating the conditional vector fields actually results in the correct vector field for modeling the marginal probability path.
Our first key observation is this:
The marginal vector field (equation 8) generates the marginal probability path (equation 6).
This provides surprising connection between the conditional VFs (those that generate conditional probability paths) and the marginal VF (those that generate the marginal probability path). This con- nection allows us to break down the unknown and intractable marginal VF into simpler conditional VFs, which are much simpler to define as these only depend on a single data sample. We formalize this in the following theorem.
Theorem 1. Given vector fields ve(x) that generate conditional probability paths Pt (ax (x(21), 1. for any distribution q(x1), the marginal vector field un in equation generates the marginal probability path pt in equation 6, i.e., u4 and pt satisfy the continuity equation (equation 25).
The full proofs for theorems all provided in Appendix B. Theorem 1 can also be derived from the Diffusion Mixture Representation Theorem in Peluchetti (2021) that provides a formula for the marginal drift and diffusion coefficients in diffusion SDEs.
3

---

Published as a conference paper at ICLR 2023
3.2 CONDITIONAL FLOW MATCHING
Unfortunately, due to the intractable integrals in the definitions of the marginal probability path and VF (equations 6 and 8), it is still intractable compute We, and consequently, intractable to naively compute an unbiased estimator of the original Flow Matching objectíve. Instead, we propose simpler objective, which surprisingly will result in the same optima as the original objective. Specifically, we consider the Conditional Flow Matching (CFM) objective,
(0) E (t.g((s)....((()))lll ve (=(221))]
where t 0 11, I1 g(xa) and now (= ). Unlike the FM objective, the CFM objective allows us to easily sample unbiased estimates as long as we can efficiently sample from pt (x) (2/(21 and compute uy (z((r)), both of which can be easily done they are defined per-sample basis. Our second key observation is therefore:
The FM (equation 5) and CFM (equation 9) objectives have identical gradients w.r.f. O.
That is. optimizing the CFM objective is equivalent (in expectation) to optimizing the FM objective. Consequently, this allows us to train a CNF to generate the marginal probability path pt-which in particular, approximates the unknown data distribution q at f=1- without ever needing access to either the marginal probability path or the marginal vector field. We simply need to design suitable conditional probability paths and vector fields. We formalize this property in the following theorem.
Theorem 2. Assuming that (x( > O for all E Rd and t E [0,1], then, up to constant independent of 0, fesm and f.u are equal. Hence, VoLrn(0) = (0Lc
CONDITIONAL PROBABILITY PATHS AND VECTOR FIELDS
The Conditional Flow Matching objective works with any choice of conditional probability path and conditional vector fields. In this section, we discuss the construction of pt(x(z1) and u, (x|=1) for a general family of Gaussian conditional probability paths. Namely, we consider conditional probability paths of the form
(=(2221) = Nx (211))
(10)
where μ : [0.1] x Rd Rd is the time-dependent mean of the Gaussian distribution, while or time-dpennden scalar standard deviation (std). We set (o 0 and oo (x1) ) = 1, so that all conditional probability paths converge to the same standard Gaussian noise distribution at t 0, p(=) (==(0, 7)- We then set lu (211) I1 and 0 (Ir)) min- which is set sufficiently small so that pı (2|=1) 1 is concentrated Gaussian distribution centered at ø1.
[0, x IR + Ry0 describes
There is an infinite number of vector fields that generate any particular probability path (e.g., by adding a divergence free component to the continuity equation, see equation 25), but the vast major- ity of these is due the presence of components that leave the underlying distribution invariant for instance, rotational components when the distribution is rotation-invarian invariant-leading to unnecessary extra compute. We decide to use the simplest vector field corresponding to canonical transforma- tion for Gaussian distributions. Specifically, consider the flow (conditioned on 211)
(x: = or (ar )ar + (x(2))
(11)
When is distributed as standard Gaussian, (xr) is the affine transformation that maps to normally-distributed random variable with mean (4r (x1) and std (21)). That to say, according to equation 4, 1/ pushes the noise distribution po(2|211) p(z) to pr(x)
[tx], p(x) = (x)
(12)
This flow then provides vector field that generates the conditional probability path:
wh (x) = un (x))))
(13)
Reparameterizing Pe (x|z1) (21) in terms of just 2o and plugging equation 13 in the CFM loss we get
( E, Br..(())))am)) (1h dt the (za)
(14)
Since dh, is a simple (invertible) affine map we can use equation 13 to solve for u, in a closed form. Let I' denote the derivative with respect time, iee., I' d f. for time-dependent function
4

---

Published as a conference paper at ICLR 2023
Theorem 3. Let pe be Gaussian probability path as in equation 10, and th its corresponding flow map as in equation 11. Then, the unique vector field that defines the has the form:
02(211) (xr - Me(a))) + (x1).
(15)
oe(x1)
Consequently, un (x( generates the Gaussian path pe (x)
4.1 SPECIAL INSTANCES OF GAUSSIAN CONDITIONAL PROBABILITY PATHS
Our formulation is fully general for arbitrary functions (x(1)) and of (Z1), ) and we can set them to any differentiable function satisfying the desired boundary conditions. We first discuss the special cases that recover probability paths corresponding to previously-usee diffusion processes. Since we directly work with probability paths, we can simply depart from reasoning about diffusion processes altogether. Therefore, in the second example below, we directly formulate a probability path based on the Wasserstein-2 optimal transport solution as an interesting instance.
Example I: Diffussin conditional VFs. Diffusion models start with data points and gradually add noise until it approximates pure noise. These can be formulated as stochastic processes, which have strict requirements in order to obtain closed form representation at arbitrary times t, resulting in Gaussian conditional probability paths pe(z((11) with specific choices of mean (x(21) ) and std (a())) (Sohl-Dickstein al., 2015; Ho al., 2020; Song et al., 2020b). For example, the reversed (noise->data) Variance Exploding (VE) path has the form
Pa Pt (1) = (2
(16)
where aa is increasing function, Øn 0, and 01 Next, equation 16 provides the choices of (11) = æ1 and or (x1) = 01 Plugging these into equation 15 of Theorem we get
(<(x)) o' x1).
(17)
The reversed (noise->datee Variance Preserving (VP) diffusion path has the form
S, B(s)ds,
- (1-a7 of-1) I), where e (r(1)))) = (x 01
(18)
and is the noise scale function. Equation 18 provides the choices of [e(21)) on and (e(11) v ar Plugging these into equation 15 of Theorem 3 get
di (01-11 x2) (1 t) u (2((z)) = 1 1- af
e (111())) - 1 T(--))
(19)
2
Our construction of the conditional VF un (x/22) does in fact coincide with the vector field previously used in the deterministic probability flow (Song et al. (2020b), equation 13) when restricted to these conditional diffusion processes; see details in Appendix D. Nevertheless, combining the diffusion conditional VF with the Flow Matching objective offers an attractive training alternative- we find be more stable and robust our experiments to existing score matching approaches.
Another important observation is that, as these probability paths were previously derived as solu- tions of diffusion processes, they do not actually reach true noise distribution finite time. In practice, (a is simply approximated by suitable Gaussian distribution for sampling and likeli- hood evaluation. Instead, our construction provides full control over the probability path, and we can just directly set μr and o,
will do next.
Example II: Optimal Transport conditional VFs. An arguably more natural choice for condi- tional probability paths is to define the mean and the std simply change linearly in time, i.e.,
(x()) txı, and oe (x)
(1 mmin) t.
(20)
According to Theorem this path is generated by the VF
Ue (x(xx)) = 21 (1 mmin )xC
(21)
(1 0min)
5

---

Published as a conference paper at ICLR 2023
1 0.0
1/o t= 4/a
1.0
1 0.0
<= 1/3
1 9/a
1.0
Diffusion path conditional score function
OT path conditional vector field
Figure 2: Compared to the diffusion path's s conditional score function, the OT path's 's conditional vector field has constant direction in time and is arguably simpler to fit with a parametric model. Note the blue color denotes larger magnitude while red color denotes smaller magnitude.
which, in contrast to the diffusion conditional VF (equation 19), is defined for all f E [0, 1]. The conditional flow that corresponds u, (2)) is
(x(r) (1 (1 - minn)t) + tz
(22)
and in this case, the CFM loss (see equations 9, 14) takes the form:
Lem(( (0) E, p( Vo (wo))) (3) - (1 Tmin mmin) )aro )|²
(23)
Allowing the mean and std to change linearly not only leads to simple and intuitive paths, but it actually also optimal in the following sense. The conditional flow (x)) is in fact the Optimal Transport (OT) displacement map between the two Gaussians pa(a))a)) and pi (x/a)). The OT
interpolant, which is a probability path, is defined to be (see Definition 1.1 in McCann (1997)):
Pt = [(1 f)id + t:p]=Po
(24)
where & : R2d -> Rdd the OT map pushing Po to Pl. denotes the identity map, i.e., id(x) x, and (1 t)id + tœ is called the OT displacement map. Example 1.7 in McCann (1997) shows, that in our case of two Gaussians where the first is standard one, the OT displacement map takes the form of equation 22.
Intuitively, particles under the OT displacement map always move in straight line trajectories and with constant speed. Figure 3 depicts sampling paths for the diffusion and OT conditional VFs. Inter- estingly, we find that sampling trajectory from diffusion paths can "overshoot" the final sample, resulting in unnecessary backtracking, whilst the OT paths are guaranteed to stay straight.
Diffusion
Figure 3: Diffusson and OT
conditional trajectories.
Figure compares the diffusion conditional score function (the re- gression target in a typical diffusion methods), i..,, V log pt (xr(211) with pt defined in equation 18, with the OT conditional VF (equation 21). The start (po) and end (p1) Gaussians are identical in both examples. An interesting observation is that the OT VF has constant direction in time, which arguably leads to simpler regression task. This property can also be verified directly from equa- tion 21 as the VF can be written in the form (z(22, shows a visualization of the Diffusion VF. Lastly, we note that although the conditional flow is opti- mal, this by no means imply that the marginal VF is an optimal transport solution. Nevertheless, we expect the marginal vector field to remain relatively simple.
((t))(xz(z1) Figure in the Appendix
5 RELATED WORK
Continuous Normalizing Flows were introduced in (Chen et al., 2018) as a continuous-time version of Normalizing Flows (see e.g., Kobyzev et (2020); Papamakarios et al. (2021) for an overview). Originally, CNFs are trained with the maximum likelihood objective, but this involves expensive ODE simulations for the forward and backward propagation, resulting in high time complexity due to the sequential nature of ODE simulations. Although some works demonstrated the capability of CNF generative models for image synthesis (Grathwohl et al., 2018), scaling up to very high dimensional images is inherently difficult. A number of works attempted to regularize the ODE to be easier to solve, e.g., using augmentation (Dupont et al., 2019), adding regularization terms (Yang & Karniadakis, 2019; Finlay al., 2020; Onken al., 2021; Tong et al., 2020; Kelly et al., 2020), or stochastically sampling the integration interval (Du et al., 2022). These works merely aim to regularize the ODE but do not change the fundamental training algorithm.

---

Published as a conference paper at ICLR 2023
SM DID
Score matching
"/ Difftsion
Flow Matching "/ Diffuusion
TM""FOT
Matching
NFT=44
Figure 4: (left) Trajectories of CNFs trained with different objectives on 2D checkerboard data. The path introduces the checkerboard pattern much earlier, while FM results in more stable training. (right) FM with OT results in more efficient sampling, solved using the midpoint scheme.
In order to speed up CNF training, some works have developed simulation-free CNF training frame- works by explicitly designing the target probability path and the dynamics. For instance, Rozen et al. (2021) consider linear interpolation between the prior and the target density but involves integrals that were difficult to estimate in high dimensions, while Ben-Hamu et al. (2022) consider general probability paths similar to this work but suffers from biased gradients in the stochastic minibatch regime. In contrast, the Flow Matching framework allows simulation-frie frre training with unbiased gradients and readily scales to very high dimensions.
Another approach simulation-free training relies the construction of diffusion process to indirectly define the target probability path (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019). Song et (2020b) shows that diffusion models are trained using denoising score matching (Vincent, 201 1), conditional objective that provides unbiased gradients with respect to the score matching objective. Conditional Flow Matching draws inspiration from this result, but generalizes to matching vector fields directly. Due to the ease of scalability, diffusion models have received increased attention, producing variety of improvements such as loss-rescaling (Song et al., 2021), adding classifier guidance along with architectural improvements (Dhariwal & Nichol, 2021), and learning the noise schedule (Nichol Dhariwal, 2021; Kingma et al., 2021). However, (Nichol & Dhariwal, 2021) and (Kingma al., 2021) only consider restricted setting of Gaussian conditional paths defined by simple diffusion processes with a single parameter- in particular, does not include conditional OT path. In an another line of works, (De Bortoli al., 2021; Wang et al., 2021; Peluchetti, 2021) proposed finite time diffusion constructions via diffusion bridges theory resolving the approximation error incurred by infinite time denoising constructions. While existing works make use of a connection between diffusion processes and continuous normalizing flows with the same probability path (Maoutsa et al., 2020b; Song et al., 2020b; 2021), our work allows us to generalize beyond the class of probability paths modeled by simple diffusion. With our work, it is possible to completely sidestep the diffusion process construction and reason directly with probability paths, while still retaining efficient training and log-likelihodd evaluations. Lastly, concurrently to our work (Liu et al., 2022; Albergo & Vanden-Eijnden, 2022) arrived at similar conditional objectives for simulation-fres training of CNFs, while Neklyudov et al. (2023) derived an implicit objective when u, is assumed to be a gradient field.
6 EXPERIMENTS
We explore the empirical benefits of using Flow Matching on the image datasets of CIFAR- 10 (Krizhevsky al., 2009) and ImageNet at resolutions 32, 64, and 128 (Chrabaszez et al., 2017; Deng et al., 2009). We also ablate the choice of diffusion path in Flow Matching, particularly be- tween the standard variance preserving diffusion path and the optimal transport path. We discuss how sample generation is improved by directly parameterizing the generating vector field and using the Flow Matching objective. Lastly we show Flow Matching can also be used in the conditional generation setting. Unless otherwise specified, using dopri5 (Dormand Prince, 1980) at absolute and relative tolerances le-5, Generated samples can be found in the Appendix, and all implementation details are in Appendix E.
evaluate likelihood and samples from the model

---

Published as a conference paper at ICLR 2023
CIFAR-10
ImageNet 32x
ImageNet 64x 64
ImageNet 128x128
Model
NLL
FIDI
NFEL
NLLL
FID.
NFE,
NLL.
FID1
NFE,
Model
NLL
FID]
Ablations
MGAN Dloeg 2,
58.9
DDPM
3.12
7.48
274
3.54
6.99
262
3.32
17.36
264
PacGANZ
57.5
Score Matching
3.16
19.94
242
3.56
5.68
178
3.40
19.74
441
Logo-GAN-AE 20ge
50.9
ScorePlow
3.09
20.78
428
3,55
14,14
195
3.36
24.95
601
Self-oond. GAN gustieal
41.7
Ours
Uncond. BigGAN ELOCCIaL 2010
25.3
FM "Y Diffusion
3.10
8.06
183
3,54
6.37
193
3.33
16.88
187
PGMGAN
21.7
OT
6.35
3.53
5.02
FM
Table 1: Likelihood (BPI), quality of generated samples (FII), and evaluation time (NFE) for the same model trained with different methods.
Score Matching w/ Diffusion
Flow Matching w/ Diffusion
Flow Matching w/ OT
Figure 6: Sample paths from the same initial noise with models trained on ImageNet 64 x4. The OT path reduces noise roughly linearly, while diffusion paths visibly remove noise only towards the end of the path. Note also the differences between the generated images.
6.1 DENSITY MODELING AND SAMPLE QUALITY ON IMAGENET
We start by comparing the same model architecture, i.e., the U-Net architecture from Dhariwal & Nichol (2021) with minimal changes, trained on CIFAR- 10, and ImageNet 32/64 with different popular diffusion-bante based losses: DDPM from (Ho et al., 2020), Score Matching (SM) (Song et al., 2020b), and Score Flow (SF) (Song et al., 2021); see Appendix E. 1.1 for exact details. Table (left) summarizes our results alongside these baselines reporting negative log-likelihood (NILL) in units of bits per dimension (BPD), sample quality as measured by the Frechet Inception Distance (FID; Heusel al. (2017)), and averaged number of function evaluations (NFE) required for the adaptive solver to reach its prespecified numerical tolerance, averaged over 50k samples, All models are trained using the same architecture, hyperparameter values and number of training iterations, where baselines are allowed more iterations for better convergence. Note that these are unconditional models. On both CIFAR-IO and ImageNet, FM-OT consistently obtains best results across our quantitative measures compared to competing methods. We are noticing a higher that usual FID performance in CIFAR- 10 compared to previous works (Ho et al., 2020; Song et al., 2020b; 2021) that can possibly be explained by the fact that our used architecture not optimized for CIFAR-IO.
Secondly, Table (right) compares a model trained using Flow Matching with the OT path on Ima- geNet at resolution 128x 128. Our FID is state-of-the-ar with the exception of IC-GAN (Casanova et al., 2021) which uses conditioning with a supervseee ResNet50 model, and therefore is left out of this table. Figures 11, 12, 13 in the Appendix show non-curated samples from these models.
Faster training. While existing works train diffusion models with a very high number of iterations (e.g., 1.3m and 10m it- erations are reported by Score Flow and VDM, respectively), we find that Flow Matching generally converges much faster. Figure 5 shows FID curves during training of Flow Matching and all baselines for ImageNet 64:64; FM-OT is able to lower the FID faster and to a greater extent than the alternatives. For ImageNet- 128 Dhariwal & Nichol (2021) train for 4.36m iter- ations with batch size 256, while FM (with 25% larger model) used 500k iterations with batch size 1.5k, i.e., 33% less image throughput; see Table 3 for exact details. Furthermore, the cost of sampling from model can dras- tically change during training for score matching, whereas the sampling cost stays constant when training with Flow Matching (Figure 10 in Appendix).
35
30
25
50
100
150
200
250
Epoch
Figure 5: Image quality during training, ImageNet 64x64.
6.2 SAMPLING EFFICIENCY
For sampling, we first draw a random noise sample 2n ~ (0, I) then compute di (xx) by solving equation with the trained VF, ve, on the interval [0, 1] using ODE solver. While diffusion
8

---

Published as a conference paper at ICLR 2023
SHEE
Edee Vilser
Edar Mipeir
Eater Hapoit
25
67
23
29
20
ODE solution
matching" "/ O/T Flow matching "/ Diffusion
Score matching
Diffusion
Figure 7: Flow Matching, especially when using OT paths, allows us use fewer evaluations for sampling while retaining similar numerical eπтоr (left) and sample quality (right). Results are shown for models trained on ImageNet 32x32, and numerical errors are for the midpoint scheme.
models can also be sampled through an SDE formulation, this can be highly inefficient and many methods that propose fast samplers (e.g., Song et al. (2020a); Zhang & Chen (2022)) directly make use of the ODE perspective (see Appendix D). In part, this is due to ODE solvers being much more efficient- sorciient-yieldingg lower enror at similar computational costs (Kloeden al., 2012) and the multitude of available ODE solver schemes. When compared to our ablation models, we find that models trained using Flow Matching with the OT path always result in the most efficient sampler, regardless of ODE solver, as demonstrated next.
Sample paths. We first qualitatively visualize the difference in sampling paths between diffusion and OT. Figure 6 shows samples from ImageNet-64 models using identical random seeds, where we find that the OT path model starts generating images sooner than the diffusion path models, where noise dominates the image until the very last time point. We additionally depict the probability density paths in 2D generation of a checkerboard pattern, Figure 4 (left), noticing a similar trend.
Low-cost samples. We next switch to fixed-step solvers and compare low (<100) NFE samples computed with the ImageNet-32 models from Table 1. Figure (left). we compare the per-pixel MSE of low NFE solutions compared with 1000 NFE solutions (we use 256 random noise seeds), and notice that the FM with OT model produces the best numerical error, in terms of computational cost, requiring roughly only 60% of the NFEs to reach the same error threshold as diffusion models. Secondly, Figure 7 (right) shows how FID changes as a result of the computational cost, where we find FM with OT is able to achieve decent FID even at very low NFE values. producing better trade- off between sample quality and cost compared to ablated models. Figure (right) shows low-cost sampling effects for the 2D checkerboard experiment.
6.3 CONDITIONAL SAMPLING FROM LOW-RESOLUTION IMAGES
Lastly, we experimented with Flow Matching for condi- tional image generation. In particular, upsampling images from 64 to 256x256.. We follow the evaluation proce- dure in (Saharia et al., 2022) and compute the FID of the upsampled validation images; baselines include reference (FID of original validation set), and regression. Results in Table Upsampled image samples are shown in Figures 14, 15 in the Appendix. FM-OT achieves simi- lar PSNR and SSIM values to (Saharia al., 2022) while
Model
FID4
ISt
PSNR
SSIM
Reference
240.8
Regression
15.2
121.1
27.9
0.801
SR3 indunseeeal 1007
5.2
180.1
26.4
0.762
FM
3.4
200.8
24.7
0.747
Table 2: Image super-resolution on the ImageNet validation set.
considerably improving on FID and IS, which as argued by (Saharia et al., (022) is a better indication of generation quality.
CONCLUSION
We introduced Flow Matching, a new simulation-free framework for training Continuous Normaliz- ing Flow models, relying on conditional constructions to effortlessly scale to very high dimensions. Furthermore, the FM framework provides an alternative view on diffusion models, and suggests forsaking the stochastic/diffusion construction in favor of more directly specifying the probability path, allowing us to, e.g., construct paths that allow faster sampling and/or improve generation. We experimentally showed the ease of training and sampling when using the Flow Matching framework, and in the future, we expect FM to open the door to allowing a multitude of probability paths (e.g., non-isotropic Gaussians more general kernels altogether).

---

Published as a conference paper at ICLR 2023
SOCIAL RESPONSIBILITY
Along side its many positive applications, image generation can also be used for harmful proposes. Using content-controlled training sets and image validatiin/classiccation can help reduce these uses. Furthermore, the energy demand for training large deep learning models is increasing at a rapid pace (Amodei al., 2018; Thompson et al., 2020). focusing on methods that are able train using less gradient updates image throughput can lead to significant time and energy savings.
ACKNOWLEDGEMEENT
We thank Qinging Zheng for her feedback. HB is supported by a grant from Israel CHE Program for Data Science Research Centers. Additionally, acknowledge the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing the core set of tools that enabled this work, includ- ing PyToorch (Paszke et al., 2019), PyTorch Lightning (Falcon & team, 2019), Hydra (Yadan, 2019), Jupyter (Kluyver et al., 2016), Matplotlib (Hunter, 2007), seaborn (Waskom et al., 2018), numpy (Oliphant, 2006; Van Der Walt al., 2011), SciPy (Jones et al., 2014), and torchdiffeq (Chen, 2018).
REFERENCES
Michael S Albergo and Eric Vanden- Building normalizing flows with stochastic inter- polants. arXiv preprint UrX1v 2022.
Dario Amodei, Danny Hernandez, Girish SastryJack, Jack Clark, Greg Brockman, and Ilya Sutskever. Ai and compute. ht https: /oppaai.
http:///pppeaaa..ccolooooo/////e ctt.cloom.bbbbccocooooooooooooaa and compute/. 2018.
Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou. curtition-guider gans. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5099-5109, 2021.
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky T. Q. Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. arXiv preprint arXiv:2207.04711, 2022.
Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance-conditioned gan. Advances in Neurul Information Processing Systems, 34227577-775299 2021.
Ricky T. Chen. torchdiffeq. 2018. torchdiffeq.
URL httpps///gtttub.ccmm/ttt////
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint /70.v0/70 2017.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnad Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. (arXiv:2106.01357), Dec 2021. doi: arXiv:2106.01357 [cs, math, stat].
211.500/rXii..10.00113333 URL http:// httt::://axii...e/ httpixxxvvv.........a www.aaiwww.....aa.aaaaaaaaaaaaae /topg///b////.... 01357.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 21L110//CVRR222005520666
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan (eds.), Advances Neu- ral Information Processing Systems, 2021. URL https::/ //ttt::////eeeevieww net wttp://wwwweeffouummeeeem fftmmorrrmmid AAWuCvzaVt.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematies, 6(1):19-26, 1980.
10

---

Published as a conference paper at ICLR 2023
Shian Du, Yihong Luo, Wei Chen, Jian Xu, and Delu Zeng. To-flow: Efficient continuous malizing flows with temporal optimization adjoint with moving speed, 2022. URL https:
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: https://proceed: ings neuri ps. cc cc./paper/201///i1ee a ott.aaacce...ea.1eeeoee-aeeeeeee pdf.
William Falcon and The PyTorch Lightning team. Pytorch lightning, 2019. URL https://
github. , com http:/Liggtnnnnggiii.a AAIIighhtii.ng.
Chris Finlay, Hern-eerik Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In ICML, pp. 3154-3164, 2020. URL hwtp: /ttp:://rooceiiieo ngs m mlr... prwss/veeveeeeeee..eee 11 9/finlay20e html.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models, 2018. URL https:
wllaxii..rgg/bbabaeee /1800 01367.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in 2020.
Neural Information Processing Systems,
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Mgan: Training generative adversarial
nets with multiple generators. In International conference on learning representations, 2018.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(3): 90, 2007.
Eric Jones, Travis Oliphant, and Pearu Peterson. {SciPy}: Open source scientific tools for (Python). 2014.
Jacob Kelly, Jesse Bettencourt, Matthew Johnson, and David K Duvenaud. Learning differential equations that easy solve. Advances in Neural Information Processing Systems, 33:4370- 4380, 2020.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In A. Beygelzimer, Y. Dauphin, Liang. and Wortman Vaughan (cds.), Advances in Neu- ral Information Processing Systems, 2021. URL https: ptttps:://ppnreviee..r net wtt.://uteeeeeoeiccoommmmo 2LdBqxc1Yv.
Peter Eris Kloeden, Eckhard Platen, and Henri Schurz. Numerical solution of SDE thEEugh computer experiments. Springer Science & Business Media, 2012.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter notebooks-a publishing format for reproducible computational workflows. In ELPUB, pp. 87-90, 2016.
Ivan Kobyzev, Simon JD Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions pattern analysis and machine intelligence, 43 (11):3964-397) 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power two samples in generative adversarial networks. Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2023
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209. 22X1:22009.0000,, 2022.
Mario Lučić, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. In International conference on machine learn- ing, pp. 4183-4192. PMLR, 2019.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker Bocker-planck equations through gradient-log density estimation. Entropy, 22(8):802, jul 2020a. doi: 10.3390/e22080802. URL https /tw.oiiiia///// org/ httgg/1..9990000
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker-
planck equations through gradient-log-densee estimation. Entropy, 22(8):802, 2020b.
Robert J McCann. A convexity principle for interacting gases. Advances in mathematics, 128(1):
153-179, 1997.
Kirill Neklyudov, Daniel Severo, and Alireza Makhzani. Action matching: A variational method
for learning stochastic dynamics from samples, 2023. URL http:///ppeeeeeeeeeeeemee wnt.tiewwweeeeeeee....... forum?id= zkhakeS.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Travis E Oliphant. Python for scientific computing. Computing Science & Engineering, 9(3): 10-20, 2007.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate contin- uous normalizing flows via optimal transport. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10))9233-9232 May 2021. URL hottps:/a //oss aaai org/ innexx.exxxe/////// .php/ AAAT/artiicle/veeww/
George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanaa. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res., 22657):1-64, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp- 8026-8037, 2019.
Stefano Peluchetti. Non-denoising forward-time diffusions. 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 06100000661..... 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: Divergence-
based generative modeling on manifolds. In A. Beygelzimer. Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL ht https: http::://ppeeeeviieeee net Eorum? 3GV 3unNJ.
Alexander Sage, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Logo synthesis and manipu- lation with clustered generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5879-5888, 2018.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image resouuiinol via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
12

---

Published as a conference paper at ICLR 2023
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distri- bution. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché-Bue, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https::/ httpss///pccoeedneneeee wwwediinnnnnnnnureeeeeeeeeeeeeee neurips. https/erriiss.ccppare/11111... wttppapeee/119//ieeee..e.e Papede pdf.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.11456,, 2020b.
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. The computational limits of deep learning. arXiv preprint arXiv:2007.05558. 2020.
Alexander Tong, Jessie Huang, Guy Wolf, David Van Dijk, and Smita Krishnaswamy. Trajectorynet: A dynamic optimal transport network for modeling cellular dynamics. In International conference on machine learning. pp. 9526-9536. PMLR, 2020.
Stefan Van Der Walt, Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing in Science & Engineering, 13(2)222, 2011.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum Wiskunde en Informatica Amsterdam, 1995.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674 2011.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning schrödinger bridge. (arXiv:2106. 10410), Jul 2021. doi: 10.48550/rXxi.2206..044 URL http: orgo abs/2105 10410. arXiv:2106. 10410 [cs].
Michael Waskom, Olga Botvinnik, Drew O' Kane, Paul Hobson, Joel Ostblom, Saulius Lukauskas, David C Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Julian de Ruiter, Cameron Pye. Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Thomas Brunner, Tal Yarkoni, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, and Adel Qalieh. mwaskom/seaborn: v0.9.0 (july 2018), July 2018. URL https: //ooi.////...// http:///oo.oo/////////// 5281/zencdo. 1313201.
Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019. URL https: / / t.ub. com http///creeerrere////y.a.aaaa
Liu Yang and George E. Karniadakis. Potential flow generator with $1_2$ optimal transport regu- larity for generative models. CoRR, abs/1908. 11462, 2019. URL http: http///xxvvv.//////eee/eeeaeelea wwwpxwiww..rr.o.ooooooooooooo abs/ 1908. 11462.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint orXiv:2204. /3902, 2022.
13

---

Published as a conference paper at ICLR 2023
A THE CONTINUITY EQUATION
One method of testing a vector field e generates probability path pt is the continuity equation (Villani, 2009). It is Partial Differential Equation (PDE) providing necessary and sufficient condition to ensuring that a vector field U/ generates pt.
d ti (x) + (z))) = 0,
(25)
where the divergence operator, div, is defined with respect to the spatial variable x (x ,..,, ie., div Ei 83
B THEOREM PROOFS
Theorem 1. Given vector fields ue (e(x(z1) that generate conditional probability paths (e (1) for any distribution g(z1), the marginal vector field ut in equation 8 generates the marginal probability path pt in equation 6, iee, us and pt satisfy the continuity equation (equation 25).
Proof. To verify this, check that Pt and un satisfy the continuity equation (equation 25);
(((()) = (a g( (dl ff diivmutolaa)nnallla))) ((
=-div muliifmmooaanie (1l (((())))()))))))) (((o)))dds)===) -div -div(u(())pu(xt)
where in the second equality used the fact that ue (1(1)) generates pe (-|ar1), in the last equality used equation 8. Furthermore, the first and third equalities are justified by assuming the integrands satisfy the regularity conditions of the Leibniz Rule (for exchanging integration and differentiation).
Theorem Assuming that (x( for all x E Rdd and E [0.1], then, up to a constant independent of 0, Lesu and Lyu are equal. Hence, VoLn((0) = VoC)
Proof. To ensure existence of all integrals and to allow the changing of integration order (by Fubini's Theorem) in the following we assume that q(x) and pe(2:(z1) are decreasing zero at sufficient speed as || + co, and that u, vt. Vgv, are bounded.
First, using the standard bilinearity of the 2-norm we have that
(x) un (x)]l = [x)] (vv(x), ux(2)) + [| (x)1 |/ (xx) ue (z/(21 )11 = 2 - un + ||,(()))))|
Next, remember that u, independent of and note that
E Ep, [1, (x)]l² = / ||l, (z))
/
)dz
= (el),, [=)111 (llx))))
where in the second equality we use equation 6, and in the third equality we change the order of integration. Next,
/ (e(x)) [u (e )da
Ep.(a) (v)( ux(x)))
pr(x)dz
p((x)
/ (c(z))) / u. (2/2z)))))) (2| (z )dz, dr
/ (o), (x(xx)) pe (a(x (a ddr (e(ee))..(((en) (x), u. (xfl=r)) where in the last equality change again the order of integration.
14

---

Published as a conference paper at ICLR 2023
Theorem 3. Let pe(za(a))) be Gaussian probability path as in equation 10, and dh its corresponding flow map as in equation 11. Then, the unique vector field that defines the has the form:
(i(m1) (ar pe(x1)) + (21)) or((r)
(15)
Consequently, (a((a))) generates the Gaussian path (e(
Proof. For notational simplicity let we (x) = 2x(x(z1). Now consider equation 1:
dt 1/ (x)) = wy (v) (x)).
Since the is invertible (as or (x1) > 0) we let x = ل (y) and get
= w.
(26)
where we used the apostrophe notation for the derivative to emphasis that ei is evaluated at sb (y) Now, inverting (x) provides
(x11
3 'r)()) or (x)
Differentiating va with respect to gives
(x ( x + (())))
Plugging these last two equations in equation 26 we get
wr(y) = o((21) ox(21)
(y) (e(xi)) + (x11
as required.
C COMPUTING PROBABILITIES OF THE CNF MODEL
We are given an arbitrary data point æ E Rd and need compute the model probability that point, i.e. Pi (I1). Below we recap how this can be done covering the basic relevant ODEs, the scaling of the divergence computation, taking into account data transformations (e.g.. centering of data), and Bits-Per-Dimension computation.
ODE for computing (z)). The continuity equation with equation 1 lead to the instantancous change of variable (Chen et al., 2018; Ben-Hamu et al., 2022):
d (og))())())) + div(v)(()(=)) = O.
dt
Integrating / E [0, 1] gives:
log (0)(x)) - log (ogoo(()(x)) = (0(((()))==
div div(rx(d
(27)
Therefore, the log probability can be computed together with the flow trajectory by solving the ODE:
vx(o)(z)) dt f(t) -iv((e((((()))
d [o((z)
(28)
Given initial conditions
()))))] f(0)
(29)
the solution (x), f(t) is uniquely defined (up to some mild conditions the VF oz). Denote
27 = ②i (x), and according to equation 27,
S(1) c log (211 log Pa(o))
(30)
15

---

Published as a conference paper at ICLR 2023
Now, we given arbitrary æ1 and want to compute (x1). For this end, will need solve equation 28 reverse. That is,
d [b) (x) ds [(1 -55]]
-0--(01--((=))
(31)
and we solve this equation for [0, with the initial conditions
(1)
x1 0
(32)
[(1)
From uniqueness of ODEs, the solution will be identical to the solution of equation 28 with initial conditions in equation 29 where log (o 1o (z,)) This can be seen from equation 30 and setting f(1) 0. Therefore we get that
f() log)o(zo) log (211)
and consequently
log (x1) = lo (x0) - f(0).
(33)
To summarize, to compute Pı (41) first solve the ODE in equation 31 with initial conditions in equation 32, and the compute equation 33.
Unbiased estimator to pi (21)) Solving equation 31 requires computation of div of VFs in Rd which is costly. Grathwohl et al. (2018) suggest to replace the divergence by the (unbiased) Hutchin- son trace estimator,
(x)- ds [ ((--s)
-01-s(01--(x)) - ( (
(34)
where E Rd is sample from random variable such that Ez2 = T. Solving the ODE in equation 34 exactly (in practice, with a small controlled error) with initial conditions in equation 32 leads to
E. [oggo(<00) f(0) lo E2 F(()) -(1)
I'' xT= D0)-o(61-
logoo( Egg
log)o(z0)
E= [2" Dvi (01-
log Po(o)) div(")-((01-d)
log Pog
(f(0) - f(1))
log (log log (R:))
log (z1).
where the third equality we switched order of integration assuming the sufficient condition of Fubini's theorem hold, and in the previous to last equality used equation 30. Therefore the random variable
Po - f(0)
(35)
is an unbiased estimator for log (æ1). To summarize, for a scalable unbiased estimation of p1 (æ1) first solve the ODE in equation 34 with initial conditions in equation 32, and then output equa- tion 35.
Transformed data. Often, before training our generative model we transform the data, e.g., we scale and/or translate the data. Such transformation is denoted by -p R + Rd and our generative model becomes a composition
(x) xp ((x))
16

---

Published as a conference paper at ICLR 2023
where R + R is the model train. Given prior probability po have that the push forward of this probability under & (equation 3 and equation 4) takes the form
pi (x)) (o(x) po( 7 (x))) det " det (0-Po(4- det (x)]
and therefore
log (ir) log Po (x)) + log det [1)]
For images = x 3 we consider a transform & that maps each pixel value from [-1, to [0, 256]. Therefore,
*(v) = 23 + 1),
and
0 (x) =2 1.
For this case we have
log (I) log d.po - 7d 2.
(36)
Bits-Per-Dimension (BPD) computation.
BPD is defined by
loga pı (x1) = Ezi
log po (x1) dlog 2
BPD =
Ez,
(37)
d
Following equation 36 we get
BPD log @=po (x dlog 2
+
and log (=po((-'(x)) approximated using the unbiased estimator in equation 35 over the trans- formed data (I1). Averaging the unbiased estimator a large test test 2ı provides a good approximation to the test set BPD.
D DIFFUSION CONDITIONAL VECTOR FIELDS
We derive the vector field governing the Probability Flow ODE (equation 13 in Song et al. (2020b)) for the VE and VP diffusion paths (equation 18) and note that it coincides with the conditional vector fields we derive using Theorem 3, namely the vector fields defined in equations 16 and 19.
We start with short primer how find conditional vector field for the probability path de- scribed by the Fokker-Planck equation, then instantiate it for the VE and VP probability paths.
Since in the diffusion literature the diffusion process from data at time I = 1, we will need the following lemma to translate the diffusion VFs to our convention of corresponds to noise and
noise time
corresponds data:
Lemma 1. Consider a flow defined by vector field u4 uxxx) generating probability density peth pe Px().) Then, the vector field (x(x) -()) generates the path pr (x) Pi P1-+(z) when initiated from Po(x) = (I))
Proof. We use the continuity equation (equation 25):
(x) (z) (2)
ddi(pt--(s)u
= -div((())((-)-((()))
and therefore (x) -m11(()) generates (x))
17

---

Published as a conference paper at ICLR 2023
Conditional VFs for Bokkerr-llance probability paths Consider Stochastic Differential Equa- tion (SDE) of the standard form
+ g,
(38)
with time parameter drift ft. diffusion coefficient gt. and dw is the Wiener process. The solution y/ to the SDE is stochastic process, i.c., continuous time-dependent random variable, the probability density of which, Pe(y)), is characterized by the Fokker-Plance equation:
dpc = div(((ap)) + Apr
9
(39)
di
2
where A represents the Laplace operator (in y). namely div where V is the gradient operator (also in y). Rewriting this equation in the form of the continuity equation can be done as follows (Maoutsa al., 2020a): dp div f.P. dt
g² VPt pt) = -div -iv((ii Vlogp.)p. = -div -div(w:)) (urpp.)
oe
2 Pt
2
where the vector field
9 v log Pt 2
ft
(40)
satisfies the continuity equation with the probability path Pc. and therefore generates Pr-
Variance Exploding (VE) path
The SDE for the VE path is
dy V d d,
where du = 0 and increasing to infinity as t > 1. The SDE is moving from data, yo. at t 0 to noise, f/, at = with the probability path
(31B0) = N of The conditional VF according to equation 40 is:
we (ylyo) of (v yo) Using Lemma we get that the probability path
(u N (vlvvn 11)
is generated by
o' (yy yo);
01
which coincides with equation 17.
Variance Preserving (VP) path
The SDE for the VP path is
T'(t) dy 2 y (tddu,
where T(1) S0 3(s)ds, E [0, 1]. The SDE coefficients are therefore
[u( (s) V, 9. (s)
and
(vll0))) = N 3 yo. (1 I).
Plugging these choices in equation 40 we get the conditional VF
(') y 1(()) yo
wi (v(v))))
(41)
2
- TT(()
Using Lemma to reverse the time we get the conditional VF for the reverse probability path:
w.(ylvo) T' t)
17(100) yo
1--e-T(11- T(1-t) y
2
(1 - t)
(11-)) tT(1 1) yo
2
e T(1-()
which coincides with equation 19.
18

---

Published as a conference paper at ICLR 2023
== 0.0
1/5
%/3
1.0
Diffusion path conditional vector field
Figure 8: VP Diffusion path's 's conditional vector field. Compare to Figure 2.
ScoreFlow
Figure 9: Trajectories of CNFs trained with ScoreFlow (Song al., 2021) and DDPM (Ho al., 2020) losses on 2D checkerboard data, using the same learning rate and other hyperparameters as Figure 4.
E IMPLEMENTATION DETAILS
For the 2D example we used an MLP with 5-layers of 512 neurons each, while for images we used the UNet architecture from Dhariwal & Nichol (2021). For images, center crop images and resize to the appropriate dimension, whereas for the 32x32 and 64x64 resolutions we use the same pre-processing (Chrabaszez al., 2017). The three methods (FM-OT, FM-Diffusion, and SM- Diffusion) are always trained on the same architecture, same hyper-parameters, and for the same number epochs.
E.I DIFFUSION BASELINES
Losses. We consider three options as diffusion baselines that correspond to the most popular dif- fusion loss parametrizations (Song & Ermon, 2019; Song al., 2021: Ho et al., 2020; Kingma al., 2021). We will assume general Gaussian path form of equation 10, i.e.
Score Matching loss is
(() (e A(t) [1))(()) lo
(42)
(e(
(n (alo, A(t) A(t)
s((x))
(43)
dx()))
Taking A(t) = of corresponds to the original Score Matching (SM) loss from Song Ermon (2019), while considering X(t) = B(1 - t) (3 is defined below) corresponds the Score Flow (SF) loss motivated by an NLL upper bound (Song et al., 2021); s, is the learnable score function. DDPM (Noise Matching) loss from Ho et al. (2020) (equation 14) is
(e(11)
L (0) E2, (x1.)
[((x)
(44)
(x(1,)
= -E po(o)) (1 (2) + mx(x))) xo
(45)
where po(a:) = (x 7) is the standard Gaussian, and e is the learnable noise function.
Diffusion path.
diffusion path we use the standard VP diffusion (equation 19), namely,
T(t) = [ (s))
(x(11) = T1.
(x(x1) V 11-af af
where - -T
19

---

Published as a conference paper at ICLR 2023
CIFARIO
ImageNet-32
ImageNet-64
ImageNet-125
Channels
256
256
192
256
Depth
3
3
3
Channels multiple
1,2,2,2
1,2,2,
1,2,3,4
1,1,2,3,4
Hcads
4
4
4
Heads Channels
64
64
64
64
Attention resolution
16
16,8
32,16,8
32,16,8
Dropout
0.0
0.0
0.0
0.0
Effective Batch size
256
1024
2048
1536
GPUs
4
16
32
Epochs
1000
200
250
571
Iterations
391k
250k
157k
500k
Learning Rate
5c-4
lc-4
lc-4
lc-4
Learning Rate Scheduler
Polynomial Decay
Polynomial Decay
Constant
Polynomial Decay
Warmup Steps
45k
20k
20k
Table 3: Hyper-parametern used for training each model
with, as suggested in Song et al. (2020b), B(s) Bmin + 8(Bmax Buuin) and consequently
T(s) S. B(r)dr simia +
1 2 3 Bmin),
where Smin 0.1, Pmax 20 and time is sampled in [0, 1 1-el, e 10-5 for training and likelihood and c = 10-5 for sampling.
Sampling. field
Score matching samples are produced by solving the ODE (equation 1) with the vector
(1 -) t)
ue (x)
2 [s((x) - x².
(46)
DDPM samples are computed with equation 46 after setting sx(x) ex(x)/or, where or
V1 af
E.2 TRAINING & EVALUATION DETAILS
We report the hyper- hyper-parameters parameters used in Table We use full 32 bit bit-precision for training CIFARIO and ImageNet-32 and 16-bit mixed precision for training 6 All models trained using the Adam optimizer with the following parameters: ,1 = 0.9, /32 = 0.999, weight decay 0.0, and 1e-8. All methods trained (i.e., FM-OT, FM-Diffusion, SM-Diffusion) using identical architectures, with the same parameters for the the same number of Epochs (see Table 3 for details). We either constant learning rate schedule
polynomial decay schedule (see Table 3). The polynomial decay learning rate schedule includes a warm-up up phase for a specified number of training steps. In the warm-up phase, the learning rate is linearly increased from 1e- to the peak learning rate (specified in Table 3). Once the peak learning rate is achieved, it linearly decays the learning rate down to 1 until the final training step.
When reporting negative log-likelibhe likeliheeood, we dequantize using the standard uniform dequantization. We report an importance-weigheer estimate using
K log pr(x + uw), where un ~ U(0, 1),
(47)
with is in {0. ..., 255} and solved at t 1 with adaptive step size solver dopri5 with atol=rtol=le- using the torchdi ffeq (Chen, 2018) library. Estimated values for different values of K are in Table 4.
20

---

Published as a conference paper at ICLR 2023
CIFAR-10
ImageNet 32x32
ImageNet 64x64
Model
K=1
K==00
K=50
K=1
K=5
K=15
K=1
K=5
K=10
Ablation
DDPM
3.24
3.14
3.12
3.62
3.57
3.54
3.36
3.33
3.32
Score Matching
3.28
3.18
3.16
3.65
3.59
3.57
3.43
3.41
3.40
ScoreFlow
3.21
3.11
3.09
3.63
3.57
3.55
3.39
3.37
3.36
Ours
FM */ Diffusion 3.23
3.13
3.10
3.64
3.58
3.56
3.37
3.34
3.33
FM */
3.11
3.01
2.99
3.62
3.56
3.53
3.35
3.33
3.31
Table 4: Negative log-likelihood (in bits per dimension) the test with different values of K using uniform dequantization.
score dif dire__ii
10a
im_dif fm
NFE
10a
100
200
300
400
500
Epochs
Figure 10: Function evaluations for sampling during training, for models trained on CIFAR-10 using dopri5 5 solver with tolerance le
When computing FID/Inception scores for CIFAR1O, ImageNet-3222 32/64 we use the TensorFlow GAN library 2. To remain comparable to Dhariwal & Nichol (2021) for ImageNet- 128 we use the evalua- tion script they include in their publicly available code repository 3.
F ADDITIONAL TABLES AND FIGURES
httpp:////ttthbb.cmm///eeeee /gui
ded- ded-di ofustio
21

---

Published as a conference paper at ICLR 2023
Figure 1I: Non-curated unconditional ImageNet-32 generated images of a CNF trained with FM- OT.
22

---

Published as a conference paper at ICLR 2023
Figure 12: Non-curated unconditional ImageNet-64 generated images of a CNF trained with FM- OT.
23

---

Published as a conference paper at ICLR 2023
Figure 13: Non-curated unconditional ImageNet- 128 generated images of CNF trained with FM- OT.
24

---

Published as a conference paper at ICLR 2023
Figure 14: Conditional generation Flow Matching OT upsampled images from
validation set.
25

---

Published as a conference paper at ICLR 2023
KINDERKLINIK
Figure 15: Conditional generation 64x64 x56x Flow Matching OT upsampled images from validation set.
26

---

Published as a conference paper at ICLR 2023
NFE=10
NFE=20
NFE=40
NFE=100
NFE=10
NFE=20
NFE=40
NFE=
Figure 16: Generated samples from the same initial noise, but with varying number of function evaluations (NFE). Flow matching with OT path trained on ImageNet-129 128.
27

---

Published as a conference paper at ICLR 2023
NF
NFE=20
NFE=40
NFE=60
NFE=100
Figure 17: Generated samples from the same initial noise, but with varying number of function evaluations (NFE). Flow matching with OT path trained on ImageNet 256
28


Reviews:
[{"review_id": "pmN0r3PRNZy", "reviewer": "Reviewer_tWC7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents flow matching, a method for training continuous-time normalizing flows by directly regressing the vector field of a chosen probability path. Flow matching only requires the ability to sample from the chosen probability path for training, and importantly does not require propagating gradients through ODE solver dynamics. The authors connect flow matching to optimal transport and existing diffusion methods for generative modeling. Empirical results suggest flow matching can give improved training and sampling efficiency over standard diffusion models. ", "summary_of_the_review": "While there are a couple of issues I would like to see addressed, I think this is a strong submission and would recommend acceptance.", "strength_and_weaknesses": "**Strengths**\nThe fundamental idea proposed in the paper is excellent. Analogously to how denoising score-matching makes score-matching tractable in diffusion models, flow matching shows that the diffusion formulation is not needed at all, and regressing the target vector field defined by a simple probability path is possible directly. The authors then explicitly lay out the connection to existing diffusion models, as well as an alternative transport path. Both the flow matching objective and the OT path are ablated independently on toy and ImageNet-level tasks. These are all strong contributions, potentially simplifying the generative modeling landscape and making strong connections in the existing literature (continuous-time flows, diffusion models, optimal transport). \n\n**Weaknesses**\nA small constant sigma_min > 0 is used throughout the paper as the initial noise applied to the data distribution. That is, t = 1 corresponds to sigma =  sigma_min instead of sigma = 0. As far as I can see, the reason for introducing sigma_min is not stated explicitly. Is it there for practical/numerical or theoretical reasons? Why can't we have sigma = 0 at t = 1? This should be explained explicitly.    \n \nI think the biggest difficulty I have with the paper is the baseline diffusion model used for the empirical evaluation. A key claim of the paper is that the proposed method effectively gives practitioners a better way to fit the vector field of a probability path than through the diffusion formulation. I would therefore expect the implementation of the diffusion baseline to be treated with due care.\n\nFor example, appendix E.1 eq (41) states that the diffusion model regresses the score function directly. In practice, diffusion models often target a quantity related to the score, such as the clean data, or the Gaussian noise added to the clean data, from which the score is derived. The choice of parameterization can significantly impact the quality of the learned model (DDPM, Ho et al. 2020), since the magnitude of the score varies widely over time. Moreover, the ODE in eq. 42 has semi-linear structure (the sum of a linear and non-linear term). Existing work (DDIM, Song et al. 2020, DPM-solver, Lu et al. 2022) has shown that using black-box solvers which don't account for this structure (i.e. solving the linear part exactly) can lead to unnecessary errors in the solution, again impacting performance. \n\nIf it is an advantage of flow matching that the parameterization of the regression target and the choice of ODE solver do not need such careful consideration, this should be stated explicitly and taken into account for direct comparison, rather than comparing to a diffusion model implementation which is not as performant as it might be. \n\nNits:\n- 4: 'There is potentially an infinite number of vector fields that generate any particular probability path,\nbut the vast majority of these' -- what is meant by vast majority? Quantify this formally. \n- 4.1: 'we can set them to any reasonable function' -- what is a 'reasonable function'? Again be precise.\n- 6.1: 'we find that we can achieve very reasonable performance' -- what is 'very reasonable performance'? \n- 6.1: The BigGAN citation is for Lučić et al. 2019 but the canonical citation is Brock et al. 2018\n- 7: 'FM can be generalized to manifold data' -- perhaps, but you can't claim this without demonstrating it ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S3t8z29BhU", "reviewer": "Reviewer_LJky", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new framework for training continuous normalizing flows, inspired by recent work on diffusion models. Originally, normalizing flows are trained by maximizing the likelihood of the pushforward density, which requires simulating the flow end-to-end for each gradient update and can thus be very costly. In this work, the proposed \"flow matching\" framework involves approximating the time-varying vector field of a desired ODE for each time t, much like the score matching algorithm in diffusion models. Thus training no longer requires end-to-end simulation of the flow, and simply involves a squared error loss for each time $t$.\n", "summary_of_the_review": "The paper is well-written and provides a straightforward generalization of the diffusion framework. Performance on ImageNet 64x64 is strong, and suggests that the theory translates well into practical improvements in image modeling.", "strength_and_weaknesses": "Strengths:\n- The work deepens the connections between diffusion models and normalizing flows originally proposed in [1], and allows for more types of probability flows, including non-diffusion flows.\n- Training involves a simple drop-in replacement of the loss in score-matching diffusion models.\n- Certain choices of the learned flow greatly improve sampling speed during inference.\n\nWeaknesses:\n- Models do not exhibit improved performance on CIFAR-10.\n- Authors claim state-of-the-art performance on ImageNet in terms of sample quality, however, FID metrics are much higher than that reported in [2] (see Table 5, for ImageNet 64x64). Can the authors comment on this?\n\n[1] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S. and Poole, B., 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.\n\n[2] Dhariwal, P. and Nichol, A., 2021. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, pp.8780-8794.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JKKKqJR_Tl", "reviewer": "Reviewer_AKwV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes flow matching, a method for training continuous normalizing flows without numerical ODE simulation. Key to flow matching is a framework for constructing suitable probability paths via a mixture of conditional (simpler) probability paths. Authors further draw connection to score-based diffusion models and optimal transport. Inspired by the connection with optimal transport, authors propose a new type of probability paths that are more efficient than existing ones in diffusion models. Experiments confirm that the proposed flow matching approach can obtain competitive image quality and likelihoods on real-world image datasets.", "summary_of_the_review": "Flow matching is a nice and clean formulation for simulation-free training of CNFs, generalizing score-based diffusion models to non-Gaussian noise and optimal transport paths. However, the idea is not completely new, and experimental results on ImageNet are unreliable.", "strength_and_weaknesses": "## Strength\n1. Very clear writing.\n2. Proposed method is simple and effective.\n3. The method generalizes conventional score-based diffusion models without complicating the formulation.\n4. The optimal transport path stabilizes training and leads to more efficient sampling.\n\n## Weaknesses\n\n1. Novelty is limited. One central contribution of this work is a method to aggregate conditional probability paths to form a target probability path that connects a simple distribution $p_0$ to the data distribution $p_1$. This is actually a special case of the result in [1]. In fact, Theorem 1 in this paper can be derived from Theorem 1 in [1] by simply replacing SDEs with ODEs (setting the diffusion coefficient to zero). In addition, the conditional flow matching objective was already proposed in [1]. In light of [1], the contributions of this work are mostly experimental verifications, and using optimal transport to construct probability paths. It is disappointing to see that this work does not cite [1] and neither does it discuss the important connections.\n\n2. Experiments on ImageNet are likely wrong. It is surprising to see that Flow Matching has a significant lead over other approaches on ImageNet 32x32 and 64x64, while having mediocre results on CIFAR-10. Even the authors' reproduction of score matching models on ImageNet is significantly better than previously reported. I'm 99% sure that this is caused by using the wrong downsampled ImageNet dataset. In generative modeling, people use the downsampled ImageNet dataset from [2], not the one from [3]. The major difference is in downsampling algorithms: the former does not use anti-aliasing and is therefore significantly more difficult for maximum likelihood training compared to the latter. Authors need to re-run their experiments on ImageNet and report the correct results.\n\n3. Authors motivate Flow Matching as a simulation-free training method for continuous normalizing flows. It is important to mention that ScoreFlow in [4] was actually proposed earlier with the same goal. In fact, ScoreFlow is the first simulation-free training method for continuous normalizing flows that are based on diffusion processes. Compared to flow matching, it has the advantage of directly maximizing log-likelihood (see also [5]) and thus will outperform flow matching in terms of NLLs. This paper should compare with ScoreFlow more carefully, include more discussions with it, and highlight aspects where flow matching outperforms ScoreFlow.\n\n## References\n[1] Peluchetti, Stefano. \"Non-Denoising Forward-Time Diffusions.\" (2021).\n\n[2] Van Den Oord, Aäron, Nal Kalchbrenner, and Koray Kavukcuoglu. \"Pixel recurrent neural networks.\" International conference on machine learning. PMLR, 2016.\n\n[3] Chrabaszcz, Patryk, Ilya Loshchilov, and Frank Hutter. \"A downsampled variant of imagenet as an alternative to the cifar datasets.\" arXiv preprint arXiv:1707.08819 (2017).\n\n[4] Song, Yang, et al. \"Maximum likelihood training of score-based diffusion models.\" Advances in Neural Information Processing Systems 34 (2021): 1415-1428.\n\n[5] Lu, Cheng, et al. \"Maximum likelihood training for score-based diffusion odes by high order denoising score matching.\" International Conference on Machine Learning. PMLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7EiyxPxKb-S", "reviewer": "Reviewer_SQzg", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Flow Matching (FM), which provides a way to scale Continuous Normalizing Flows (CNFs) to very high dimensions such as ImageNet-128. Rather than using expensive simulation-based training approaches (as for conventional CNF training), the authors propose to regress the vector field learned by a neural network against the true vector field given by predefined conditional probability paths. This approach for leveraging conditional probability paths, and also using optimal transport (OT) displacement maps for Gaussian conditional probability paths, not only leads to computational gains, but also provides an alternative perspective on diffusion-based generative models that improves their performance and stability as well.", "summary_of_the_review": "Flow Matching is an interesting approach for scaling and improving the performance of not only CNFs, but also conventional diffusion-based generative models. The idea of matching vector fields and defining the generative process in terms of intuitive probability paths allows for not only empirical improvements in CNFs, but also ways to go beyond pre-defined diffusions and data modalities (e.g. this could be used for manifold data). This contribution is valuable and should be highlighted at the conference.", "strength_and_weaknesses": "Strengths:\n- The empirical results are very strong: CNFs trained with OT-based FM achieves SOTA on likelihood estimation relative to existing baselines (though some of these competing methods do not actually compute likelihoods on that scale) and sample quality in terms of FID on ImageNet-{32,64}. They are competitive with existing baselines on CIFAR-10.\n- OT-based FM also leads to more stable training, faster model convergence, and the simple/intuitive way in which the probability paths are defined leads to desirable properties such as “faster” (more efficient) generation of the desired sample starting from random noise. This is in contrast to diffusion models, where the perceptual quality of the denoised (generated) sample increases drastically towards the latter timesteps.\n- Theoretically, the approach provides a nice perspective for understanding and unifying diffusion-based generative models. And by matching vector fields, this approach allows for the generalization of such generative models beyond the class of probability paths modeled by simple diffusions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%