Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
Authors: Xingchao Liu
Chengyue Gong
qiang liu
Venue: ICLR 2023 notable top 25% (2023)

Abstract: We present rectified flow, a simple approach to learning (neural) ordinary differential equation (ODE) models to transport between two empirically observed distributions $\pi_0$ and $\pi_1$, hence providing a unified solution to generative modeling and domain transfer, among various other tasks involving distribution transport. The idea of rectified flow is to learn the ODE to follow the straight paths connecting the points drawn from $\pi_0$ and $\pi_1$ as much as possible. This is  achieved by solving a straightforward nonlinear least squares optimization problem, which can be easily scaled to large models without introducing extra parameters beyond standard supervised learning. The straight paths are the shortest paths between two points, and can be simulated exactly without time discretization and hence yield computationally efficient models. We show that, by learning a rectified flow from data, we effectively turn an arbitrary coupling of $\pi_0$ and $\pi_1$ to a  new deterministic coupling with provably non-increasing convex transport costs. In addition, with a ``reflow" procedure that iteratively learns a new rectified flow from the data bootstrapped from the previous one, we obtain a sequence of flows with increasingly straight paths, which can be simulated accurately with coarse time discretization in the inference phase. In empirical studies, we show that rectified flow performs superbly on image generation, image-to-image translation, and domain adaptation. In particular, on image generation and translation, our method yields nearly straight flows that give high quality results even with \emph{a single Euler discretization step}. Code is available at \url{https://github.com/gnobitab/RectifiedFlow}.

Published as a conference paper at ICLR 2023
FLOW STRAIGHT AND FAST: LEARNING TO GENER- ATE AND TRANSFER DATA WITH RECTIFIED FLOW
Xingchao Liu", Chengyue Gong, Qiang Liu Department of Computer Science
University of Texas at Austin
[xcliu, cygong, (qiang)ecs cec.ss.ueeeeeee..
ABSTRACT
We present rectified flow, simple approach to learning (neural) ordinary differen- tial equation (ODE) models to transport between two empirically observed distri- butions No and T1 hence providing unified solution to generative modeling and domain transfer, among various other tasks involving distribution transport. The idea of rectified flow is learn the ODE to follow the straight paths connecting the points drawn from To and πi as much as possible. This is achieved by solv- ing a straightforward nonlinear least squares optimization problem, which can be easily scaled to large models without introducing extra parameters beyond stan- dard supervised learning. The straight paths are the shortest paths between two points, and can be simulated exactly without time discretization and hence yield computationally efficient models. We show that, by learning a rectified flow from data, we effectively turn arbitrary coupling of roo and π to new deterministic coupling with provably non-increasing convex transport costs. addition, with a "reflow" procedure that iteratively learns strapped from the previous one, we obtain a sequence of flows with increasingly straight paths, which can be simulated accurately with coarse time discretization in the inference phase. In empirical studies, we show that rectified flow performs superbly on image generation and image-to-image translation. In particular, on image generation and translation, our method yields nearly straight flows that give high quality results even with single Euler discretization step. Code available at htt / //gtttgttbb//////////// com/ ggpbgnnbitttttto tabbtt/a/aaa /aaatt///iiii.eecceeeeeeeeceee.e FiedFlow.
rectified flow from the data boot-
INTRODUCTION
Compared with supervised learning, the shared difficulty of various forms of unsupervised learning is the lack of paired input/output data that makes standard regression or classification tasks possible. The crux of many unsupervised methods to find meaningful correspondences between points from two distributions. For example, generative models such generative adversarial networks (GAN) and variational autoencoders (VAE) (e.g., Goodfellow et al., 2014; Kingma & Welling, 2013; Dinh et al., 2016) seek to map data points to latent codes following simple elementary (e.g., Gaussian) distribution with which the data can be generated and manipulated. On the other hand, domain transfer methods find mappings to transfer points between two different data distributions, both observed empirically, for the purpose of image-to- amage-to-image to-image translation, style transfer, and domain adaption (e.g., Zhu al., 2017; Flamary al., 2016; Trigila & Tabak, 2016; Peyré et al., 2019). These tasks can be framed unifiedly as finding a transport map between two distributions:
Learning Transport Mapping Given empirical observations of two distributions no, Ti on Rd, find a transport map T; Rdd + Rd, which, in the infinite data limit, gives Zi := T(Zo) ~ ग when Zo ~ To, that (Zo, Zi) is coupling (a.k.a transport plan) of no and क.
We should note that the answers of this problem are not unique because there are often infinitely many transport maps between two distributions. Optimal transport (OT) (c.g.. Villani, 2021; Am- brosio et al., 2021: Figalli & Glaudo, 2021; Peyré et al., 2019) addresses the more challenging
Xingchao and Chengyue contributed equally.

---

Published as a conference paper at ICLR 2023
Initialization
N
N-2
5
N- 10
N 1000
Distilled (N
I.Rectifeed Flow
2.Reciffed Flow
I-Rectified Flow
2.Restified Flow
Figure 1: The trajectories of rectified flows for image generation (Np: standard Gaussian noise, W1. cat faces, top two rows), and image transfer between human and cat faces (so: human faces, wi: cat faces, bottom two rows), when simulated using Fuler method with step size 1/M for N steps. The first rectified flow induced from the training data (I-rectified flow) yields good results with very small number (e.g.. 2 2) steps; the straightened reflow induced from --rectified flow (denoted as 2-rectified flow) has nearly braight line trajectories and yield good results even with one discretization step.
problem of finding an optimal coupling that minimizes a notion of transport cost, typically of form
Blc(Z Zo)], where Rd R is cost function, such c(x) ||a|| However, for the gen- erative and transfer modeling tasks above, the transport cost is not of direct interest, even though induces a number of desirable properties. Hence,
not necessary to accurate solve the OT problems given the high difficulty of doing so. An important question is to identify relaxed notions of optimality that are of direct interest for ML tasks and are easier enforce in practice.
Several lines of techniques have been developed depending on how to represent and train the map T. In traditional generative models, T parameterized as a neural network, and trained with either GAN-type minimax algorithms or (approximate) maximum likelihood estimation (MLF). However, GANs suffer from numerically instability and mode collapse issues, and require substantial engi- neering efforts and human tuning. which tend to transfer poorly across different model architecture and datasets. On the other hand, MLE tends to be intractable for complex models, and hence requires either approximate (variational Monte Carlo) inference techniques such as those used in VAE, or special model structures that yield tractable likelihood such as normalizing flow and auto-regressive models, which causes difficult trade-offss between expressive power and computational cost.
Recently, advances have been made by representing the transport plan implicitly as a continuous time process, including flow models with neural ordinary differential equations (ODEs) (e.g., Chen al., 2018; Papamakarios et al., 2021; Song et al., 2020a) and diffusion models by stochastic differential equations (SDEs) (e.g., Song et al., 2020b: Ho et al., 2020; Tzen & Raginsky, 2019; De Bortoli al., 2021; Vargas et al., 2021). In these models, neural network is trained to represent the drift force of the processes and numerical ODE/SDE solver is used to simulate the process during inference. By leveraging the mathematical structures of ODEs/SDEs, the continuous-time models can be trained efficiently without resorting minimax or traditional approximate inference techniques. The most notable examples are the score-based generative models (Song & Ermon, 2019; 2020; Song et al., 2020b) and denoising diffusion probabilistic models (DDPM) (Ho al., 2020), which has achieved impressive empirical results image generation recently (e.g.. Dhariwal & Nichol, 2021). How- ever, compared with the traditional "one-step" models like GAN and VAE, continuous-time models effectively "infinite-step" and cast high computational cost in inference time: drawing a single point (e.g., an image) requires to solve the ODE/SDE with a numerical solver that needs to repeat- edly call the expensive neural force field for large number of times. Moreover, in existing approaches, generative modeling and domain transfer are typically treated separately. It often requires to extend techniques to solve domain transfer problems; see e.g., Cycle
2

---

Published as a conference paper at ICLR 2023
X
X
X (7a)
2
Rectified Flow
Linear
Interpatation
(a) Lincar interpolation (b) Rectified flow Z,
(c) Linear interpolation
(d) Rectified flow Z
X, t.x (1 e) induced by X1)
2. 12: + 2) induced by (2a, 21)
Figure 2: (a) Linear interpolation of data input (i) ~ No x æι. (b) The trained rectified flow Ze the trajectories are "rewired" at the intersection points to avoid crossing. (c) The lincar interpolation the end points (o Z1) of flow Ze- (d) The rectified flow induced from (Zo, Z1), which follows straight paths.
GAN (Zhu et al., 2017) and diffusion- based saage-to-image translation (e.g., Su et al., 2022; Zhao et al., 2022). One framework that naturally unifies both tasks is optimal transport, which, however, is challenging to solve for problems with high dimensional and large volumes of data.
Contribution We introduce rectified flow, a surprisingly simple approach to the transport mapping problem, which unifiedly solves both generative modeling and domain transfer. The rectified flow is an ODE model that transport distribution To to πi by following straight line paths as much as possible. The straight paths are preferred both theoretically because is the shortest path between two end points, and computationally because it can be exactly simulated without time discretization. Hence, flows with straight paths bridge the gap between one-step and continuous-timg models.
Algorithmically, the rectified flow is trained with simple and scalable unconstrained least squares optimization procedure, which avoids the instability issues of GANs, the intractable likelihood of MLE methods, and the subtle hyper-parameter decisions of denoising diffusion models. The proce- dure of obtaining the rectified flow from the training data has the attractive theoretical property of 1) yielding coupling with oon-increasing transport cost jointly for all convex cost c, and 2) making the paths of flow increasingly straight and hence incurring lower error with numerical solvers. With re- flow procedure that iteratively trains new rectified flows with the data simulated from the previously obtained rectified flow, we obtain nearly straight flows that yield good results even with very coarse time discretization. Our method is purely ODE-based, both conceptually simpler and practically faster in inference time than the SDE-based methods (Ho et al., 2020; Song et al., 2020b;a).
As shown in Figure 1, rectified flow yields high-quality results for image generation when simulated with a very few number of Euler steps (see Figure 1, top row). Moreover, with just one step of reflow, the flow becomes nearly straight and hence yields good results with one Euler discretization step (Figure 1, the second row). This substantially improves over the standard denoising diffusion methods. Quantitatively, we claim a state-of-the-an result of FID (4.85) and recall (0.51) on CI- FARIO for one-step fast diffusion/floww models (Bao al., 2022: Lyu al., 2022; Xiao al., 2021; Zheng et al., 2022; Luhman & Luhman, 2021). The same algorithm also achieves superb results on the image-to-image translation task (see the bottom two rows of Figure 1).
2
METHOD
We provide an overview of the method in Section 2.1 with more discussion in Section 2.2.
2.1
OVERVIEW
Rectified flow Given empirical observations of Xo ~ To and X2 ~ WT, the rectified flow induced from (Xo, X1) an ordinary differentiable model (ODE) on time E [0, 1).
dz, = o(Ze, t)dt.
which converts Zo from To to a Z1 following Ti. The drift force v: Rd + Rd is set to drive the flow to follow the direction (X, - Xx) of the linear path pointing from X, X, as much as possible, by solving a simple least squares regression problem:
min / E
[|(( Xo) vXe, 0)1²] dt, with X, = + (1 tt)
(I)
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 Rectified Flow: Main Algorithm
Procedure: Z = RectFlou( (Xe, X1)):
Inputs: Velocity model uo: E + R" with parameter 0.
Training: arg min I.. [[]X; Xo v(tx1 + (1 X), )) with Uniform [0. 1])-
8
Sampling: Draw (Zo. Z1) following dZt 1/a (Ze: t)dd starting from Zo so (or Zi ~ TT).
Return: Z= { 7z E [0, 1]].
Reflow (optional): z++1 RectFlou((65 2))). starting from (28, 2 = (Xo, X1), where (Xo, X1)
drawn from No and ni.
Distill (optional): Learn a neural network 7 to distill the k-rectified flow, such that Z ~ ((z))
where X, is the linear interpolation of X, and X1. The expectation E[|] here is w.r.t. the randomness
of (Xo, X1) while treating X; t.x1 + (1 - Xx,, as function of (Xu. X1). To understand the method intuitively, note that the linear interpolation X, follows an naive ODE of dX, = (X1 Xo)dt. This ODE not practically useful for constructing a transport map as is non-causal (or anticipating): the update of X, requires the information of the final point X1- By fitting the drift e with X1 Xo. the rectified flow causalizes the paths of linear interpolation Xt, yielding an ODE flow that can be simulated without seeing the future.
In practice, we parameterize v with neural networks or other nonlinear models and solve (1) with any off shel stochastic optimizer, such stochastic gradient descent, with empirical draws of (XoxX)), After we get v, we solve the ODE either forwardly starting from Zo ~ To to transfer To to mi backwardly from Zı Ti to Zo To. Specifically, for backward sampling, we simply solve dxx, -u -x,X,, t) initialized from X,, TT and set X, = X1_ The forward and backward sampling are equally favored by the training method, because the loss in (1) is timesyymmetric in that it yields the equivalent problem when exchanging Xo and X and flipping the sign of v.
Flows avoid crossing A key to understanding the method is the non-crossing property of flows as illustrated in Figure 2: the different paths of well-defined ODE az, = v v1 (4,,,)dl, t)dl, whose solution exists and is unique, cannot cross each other at any time E [0, 1). Specifically, there exists no location z Rdd and time E 10, such that two paths go across z at time along different directions, because otherwise the solution of the ODE would be non-unique. On the other hand, the paths of the interpolation process X, may intersect with each other (Figure 2a), which what makes it non-causal. Hence, as shown in Figure 2b, by solving the optimization in (1), the rectified flow rewires the individual trajectories passing through the intersection points to avoid crossing, while tracing out the same density map as that of the linear interpolation paths. We can view the linear interpolation X2 as building roads (or tunnels) that connect To and Ti, and the rectified flow as traffics of particles passing through the roads in a myopic, memoryless, non-crossing way, which ignore the global path information of how X, and X, are paired, and rebuild a more deterministic pairing of (Zo, Z1).
Rectified flows reduce transport costs If (1) is solved exactly, the pair (Zu, Z1) of the rec- tified flow is guaranteed to be valid coupling of Ta, T (Theorem D.3), that is, Zi follows mi if Zo ~ Tp- Moreover, (Zo, Zi) guarantees to yield no larger transport cost than the data pair (Xo.X1) X1) simultaneously for all convex cost functions e (Theorem D.5). The data pair (Xo, X1) can be an arbitrary coupling of To. TT. typically independent (i.c., X1) ~ To X T1) as dic- tated by the lack of meaningfully paired observations in practical problems. In comparison, the rectified coupling (Zo. Z1) has deterministic dependency as it is constructed from an ODE model. Denote by (Zu, Z₁) = DrectFlow((x,) X1)) the "rectification" map from (Xu, X1) to (Zo, Z1). Hence, RectFlow(-) converts an arbitrary coupling into deterministic coupling with lower convex transport costs.
Straight line flows yield fast simulation Denote by Z = ReectFlow((X,,, X1)) the rectified flow induced from (Xo, X1). Applying this operator recursively following Algorithm yields a sequence of rectified flows Z g++1 = RectFlow((Z] Z))) with (2 = (Xo, X1), where zt is the k-th rectified flow, or simply k-rectified flow, induced from (Xo- X1)
This reflow procedure not only decreases transport cost, but also has the important effect of straight- ening the paths of rectified flows, that is, making the paths of the flow more straight. This is highly
4

---

Published as a conference paper at ICLR 2023
attractive computationally as flows with nearly straight paths incur small discretiretinninnnn error in numerical simulation. Indeed. perfectly straight paths can be simulated exactly with a single Euler step and is effectively one-step model. This addresses the very bottleneck of high inference cost in existing continuous-time ODE/SDE models.
2.2 MAIN PROPERTIES
We provide more in-depth discussions the main properties of rectified flow. We highlight the intuitions in this section and defer the full course theoretical analysis to Appendix D.
First, for given input coupling (Xo, X1), the exact minimum of (1) is achieved if
(2) which is the expectation of the line directions X, Xo that pass through x at time We discuss below the property of rectified flow dz, xx (Ze, t)dt with Zo To, assuming that the ODE has an unique solution.
xxx (ax t) = B[X1 - Xo Xx = x].
Marginal preserving property [Theorem D.3] The pair (Zo Zi) is a coupling of and The marginal law of Zy equals that of X₁ at every time I, that is, Law (2) Law VI E [0, 1).
Intuitively, this holds because, by the definition of x in (2), the expected amount of mass that passes through every infinitesmal volume at all location and time are equal under the dynamics of X, and Z,, which ensures that they trace out the same marginal distributions:
Flow in & out
Flow in & out
Vtime & location = Law(Zi) Law(Xx) Vt.
Mathematically, this results is proved in Section D.1 by showing that the marginal laws of both X, and Z4 satisfy the same continuity equation Pr + V (v- pc) 0, and are hence equivalent.
On the other hand, the joint distributions of the whole trajectory of Ze and that of X are different in general. In particular, X, is in general a non-causal, non-Markov process, with (Xo, Xi) a stochastic coupling, and Ze cansalizes, Markovianizes and derandomizes X,, while preserving the marginal distributions at all time.
Reducing transport costs [Theorem D.5] The coupling (Zo, Z1) yields no larger convex transport costs than the input (Xo, X1) in that E[c(Z, Zo)] s B[c(X1 Xo)] for any convex e: R2 + IR.
The transport costs measure the expense of transporting the mass of one distribution to another fol- lowing the assignment relation specified by the coupling and a central topic in optimal transport (e.g.. Villani, 2009; 2021; Santambrogio, 2015; Peyré al., 2019; Figalli & Glaudo, 2021). Typical examples are c(.) = 1--1 with a 2 1. Hence, RectFlow(-) yields Pareto descent on the collection of all convex transport costs, without targeting any specific This distinguishes it from the typical optimal transport optimization methods, which are explicitly framed to optimize a given c. As a result, recursive application of RectFlow(-) does not guarantee attain the c-optimal coupling for any given with the exception in the one- me-dimensional case when the fixed point of RectFl ow(() co- incides with the unique monotonic coupling that simultaneously minimizes all non-negative convex costs c; see Appendix D.4.
Intuitively, the convex transport costs are guaranteed decrease because the paths of the flow Z, is rewiring of the straight paths connecting (Xo, X1). To give an illustration, consider the simple case of c(.) = ||-I| when transport costs [[[] Xn- - X, and B[IlZo - Zn Il] are the expected length of the straight lines connecting the end points. The inequality can be proved graphically as follows:
( Length
El 21 Length
Length
Ell| Xill],
(+)
where < uses the triangle inequality, and straight paths of X,, following the construction of x in (2). For general convex c, a similar proof using Jensen's 's inequality shown in Appendix D.2.
holds because the paths of Z, is a rewiring of the
Reflow, straightening, fast simulation As shown in Figure 3, when we recursively apply the
procedure Z RectFloww (Zz) 2))), the paths of the k-rectified flow z increasingly
5

---

Published as a conference paper at ICLR 2023
Itansport Straightnes:
30
Sellow Step
(a) Ist rectified flow z'
(b) Reflow Z
(c) Reflow z
Transport cost,
z RectFloo((X X1)) z RactFlou (/) 71))
z RectFlou((2] Zi))
Straightness
Figure 3: (a)-(c) Trajectories of the reflows toy example (To: purple dots, red dots; the green and blue lines trajectories connecting different modes of mo, #T). (d) The straightness and the relative 1.2 transport cost v.s. the reflow steps. See Appendix D.5 for more information.
straight, and hence easier to simulate numerically, as k increases. This straightening tendency can be guaranteed theoretically.
Specifically, we say that a flow dZz = o(Z,, is straight if we have almost surely that Z₁ tZ (1 Zo for Vt E [0, equivalently v(Z,,t Zi Zo const following each path. (More precisely, "straight" here refers to straight with a constant speed.) Such straight flows highly attractive computationally
it is effective one-step model: single Fuler step update Z1 = Zo v( Zo. 0) calculates the exact Z from Zo. Note that the linear interpolation X = {Xx, is straight by this definition but is not a (causal) flow and hence can not be simulated without an oracle assess to draws of both To and T1. In comparison, it is non-trivial to make a flow dZz = zZ t)dt straight, because if so v must satisfy the inviseid Burgers' equation O. + (0_0)u- =
d 1) (2. =
0,u( (Z,,)) 02v( Z,, ()o( (z. 0.u( 1) 0.
dt
More generally, the straightness of any smooth process Z = {Zz} can be measured by
S(Z) / E [1 Zo) 2, |9 dt.
|
S(Z) = 0 means exact straightness. A flow with small S has nearly straight paths and hence can be simulated accurately using numerical solvers with a small number of discretization steps. Appendix D.3 shows that applying rectification recursively provably decreases S(Z) towards zero.
[Theorem D.7] Let Zt be the k-th rectified flow induced from (Xo, X1). Then
minke(o-- [J S(Z*) < -X K
As shown Figure applying one step of reflow can already provide nearly straight flows that yield good performance when simulated with a single Euler step. It is not recommended to apply too many reflow steps as it may accumulate estimation error on u.
Distillation After obtaining the k-th rectified flow zt, we can further improve the inference
speed by distilling the relation of (2) 2 into a neural network π directly predict 2 from Z without simulating the flow. Given that the flow is already nearly straight (and hence well approximated by the one-step update), the distillation can be done efficiently. In particular, if we take
((=)) = zo + v(20, 0), then the loss function for distilling z is E [|(2) 2$) v(z 0)]"
which is the term in (1) when t = 0. The difference between distillation and rectification should be highlighted: distillation attempts to faithfully approximate the coupling (Z) ZT).. while rectification yields different coupling ( 7 with lower transport cost and more straight flow. Hence, distillation should be applied only in the final stage for fine-tuning the model to one-step inference.
Nonlinear rectified flow and friends (Appendix C) Our method can be extended to a simple yet high general nonlinear rectified flow method which uses any smooth interpolation curve X. between Xo and X₁ and training loss of fo E[|]- v(X,, ]d], where X, θX,, is the time derivative. A natural class of Xt, which include probability flow ODEs (Song et al., 2020b) and DDIM (Song et al., 2020a) as the special cases, is X, x X,X,, for some a, 3r sequences satisfies (approximately) ag = 3 ar. Sr the X, may not have straight trajectories and can not yield straight flows even with reflows.
and og = S1 = O. However, depending on the choices of
Interestingly, the idea of learning generative ODEs with interpolating curves (c.g., 1-rectified flow with general X,) was concurrently proposed in number of works (Lipman et al., 2022; Albergo &

---

Published as a conference paper at ICLR 2023
GAR
Mantel
1017
Recalli
foledAo Wome 3020
Milas 2010
0.8
20.3
044
133
1643
2-Mantthet Blow tedtust?
E-lntiieed Blow (ellish
12:21 (443)
0:34 03.39
MaaiioN Fe (laentt d 7000
8:11
6.18
0.33
ODE (eflast))
(reat)
1.29
(16 27)
0.0:9.59
suh VP ODE (Seg et al. 30000) (-Dunil)
1.21 68.80)
431
(14
0.099.35
Dapasin Cata Oiancial 2020
800
145
Piat Seaslaton PRaue (2C0) 1.taamm
FRestina Flow
127
X:89
2.58
1337
5:24
NCS comber Co
1.1
a6 1214
60000
VP ont (Sngaat N000
3-Bettikll
100
9:37
145
0.51
DDM Seng mil 3050
548
cB-VP Sobe in 300B)
196
9/46
166
50E
Fal Sennibrna (Saie mares N00000
(Sng
(Seg
2000
NLN VV Serreeepc (200)
1:0
FAR www.daittteereeoo www.weeeeeeeerrre.oo
JoNL cal. 113
190
(a) Results using the DDPM+ architecture.
(b) Recent results with different architectures reported in literature.
Table 1: Results on CIFAR10 10 unconditioned image generation. Fréchet Inception Distance (FID) and Inception Score (IS) measure the quality of the generated images, and recall score (Kynkäänniemi et al., 2019) measures diversity. The number of function evaluation (NFE) denotes the number of times we forward through the main neural network. It coincides with the number of discretization steps N for ODE and SDE models.
Vanden-Eijnden 2022; Heitz et al., 2023). Neklyudov et al. (2022) proposes a different but highly related method for the problem of trajectory inference from uncorrelated samples (e.g.. Hashimoto et al., 2016; Lavenant et al., 2021). The distinctive features of our work include enabling fast, one-step generation via ODEs with straight trajectories using the reflow procedure, and the clari- fied connections to optimal transport (OT) shown in Appendix D, which is further elaborated in a companion work (Liu, 2022a). A different connection to OT was discussed in Albergo & Vanden- Fijnden (2022) which proposes a minimax procedure that yields L2 optimal transport maps.
3 EXPERIMENTS
1-Sesiifill
Setup We follow the procedure in Algorithm We start with drawing (Xo, X1) no X Ti and use it to get the first rectified flow zt by minimizing (1). The second rectified flow z² is obtained by the same procedure except with the data replaced by the draws from (Z). Z1). ). generated by simulating the first rectified flow z This process is repeated for k times get the k-rectified flow zt Finally, we further distill the k-rectified flow Zt into one step model 2 = z0 + v(zo. 0) by fitting it on draws from (Z). ). By default, the ODEs are simulated us- ing the vanilla Euler method with constant step size 1/N for N steps, that is, 2 = 2, 7 o (2, t)/N for t E {o...., NY/N. We use the Runge- Kutta method of order 5(4) from Scipy (Virtanen al., 2020), denoted as RK45, which adaptively decide the step size and number of steps N based on user-specified relative and absolute tolerances. In our experiments, we stick to the same parameters Song al. (2020b). More details can be found in Appendix E.
vab-VP
2-Renified
0.01
0.025
0.5
Figure 4: Sample trajectories 2t of different flows on the AFHQ Cat dataset, and the extrapolation zf 2t + (1 t))((z, t) from different 2;. The same random seed adopted for all methods. The of of 2-rectified flow almost independent with 1. iindictting that its trajectory is almost straight.
3.1 UNCONDITIONED IMAGE GENERATION
Experiment settings For generative modeling, we set No to be the standard Gaussian distribution and n the data distribution. Our implementation of rectified flow is modified upon the open-source code of (Song et al., 2020b). We adopt the U-Net architecture of DDPM++ (Song et al., 2020b) for representing the drift x and report in Table 1 (a) and Figure 5 the results of our method and the (S ODE from Song et al. (2020b) using the same architecture. Other recent results using different network architectures are shown in Table 1 (b) for reference.
Results Results fully solved ODEs. As shown in Table (a), the 1-rectified flow trained on the DDPM++ architecture, solved with RK45, yields the lowest FID (2.58) and highest recall (0.57) among all the ODE-based methods. In particular, the recall of 0.57 yields a substantial improvement over existing ODE and GAN methods.
7

---

Published as a conference paper at ICLR 2023
-O O- 1POIT- 12 OUE SCC-L---SHL-KK- VP
[x=2Xx03
- Livetted Flw C 1 Besther Hln C Meethee Libette thute Letitee
Batten
Biller
Bilter
0.5
seccall
0.48
F10
0.42
30 100
1000
30
300
1000
20
Number Staps (log)
Number Steps [og)
Trabning
(b) FID and Recall vs. Training Iterations Figure 5: (a) Results of rectified flows and (Sub-)VP ODE on CIFARIO with different number N of Euler discretization steps. (b) The FID and recall during different reflow and training steps. In (a), k- (-Distilled refers to the one-step model distilled from k- R-Rectified Flow for k = 1,2,3.
(a) FID and Recall vs. Number of Fullr discretization steps N
Results on few and single step generation. As shown in Figure 5, the reflow procedure substantially improves both FID and recall in the small step regime (e.g., .N 80), even though it worsens the results in the large step regime due to the accumulation of eπrоr show that each reflow leads to a noticeable improvement in FID and recall. For one-step generation (N = 1), the results are further boosted by distillation (see the stars in Figure 5 (a)). Overall, the distilled k-Rectified Flow with & = 1, 2, 3 yield one-step generative models beating all previous ODEs with distillation; they also beat the reported results of one-step models with similar U-net type architectures trained using GANs (see the GAN with U-Net in Table 1 (b)). In particular, the distilled 2-rectified flow achieves an FID of 4.85, beating the best known one-step generative model with U-net architecture, 8.91 (TDPM, Table I (b)). The recalls of both 2-rectified flow (0,50) and 3-rectified flow (0.51) outperform the best known results of GANs (0.49 from StyleGAN2+ADA) showing an advantage in diversity.
estimating v², Figure 5 (b)
Refler
Refow
RRettfed Flew 3 Racified Flw
Reflow straightens the flow. Figure shows the reflow procedure decreases improves the straight- ness of the flow on CIFARTO. In Figure 4 visual- izes the trajectories of 1 l-rectified flow and 2-rectified flow on the AFHQ cat dataset: at each point 2,, we extrapolate the terminal value at t
0.061
graigheeess
3.050
A-eetiled Par
by of 2 + (1 (), t); if the trajectory of ODE fol- vary t when following the same path. We observe that 24 is almost independent with / for 2-rectified flow, showing the path is almost straight. Moreover, even though 1-rectified flow is not straight with of over time, it still yields recognizable and clear images very early (t ~ 0.1). In comparison, it is need t F 0.6 to get clear image from the extrapolation of sub- VP ODE.
0.057
10
20
30
90
30
80
Number Treining Iterations (x10")
lows a straight line, 21 should not change
Figure The straightening effect on CIFARIO. Left: the straightness measure w.r.t. reflow steps and iterations. Right: trajectories of randomly sampled pixels following 1- and 2-rectified flow.
High-resolution image generation Figure 7 shows the result of 1-rectified flow on image gener- ation on high-resolution (256 x 256) datasets, including LSUN Bedroom (Yu et al., 2015), LSUN Church (Yu et al., 2015), CelebA HQ (Karras et al., 2018) to AFHQ Cat (Choi et al., 2020). Ours can generate high quality results across the different datasets. Figure 4 show that 1-(2-) rectified flow yields good results with one or few Euler steps.
3.2 IMAGE-TO-IMAGE TRANSLATION
Assume we are given two sets of images of different styles (a.k.a. domains), whose distributions are denoted by No. Ti, respectively. We are interested in transferring the style (or other key char- acteristics) of the images in one domain to the other domain, in the absence of paired examples. A classical approach to achieving this is cycle-consistent adversarial networks (a.k.a. CycleGAN) (Zhu et al., 2017: Isola et al., 2017), which jointly learns a forward and backward mapping F, G by minimizing the sum of adversarial losses on the two domains, regularized by a cycle consistency loss to enforce F(G(z)) = x for all image x. By constructing the rectified flow of Tu and T1, we obtain simple approach image translation that requires no adversarial optimization and cycle- consistency regularization: training the rectified flow requires a simple optimization procedure and the cycle consistency is automatically in flow models satisfied due to reversibility of ODEs.
8

---

Published as a conference paper at ICLR 2023
(A) LSUN Church
(B) CelebA HQ
(C) Bedroom
AFH Cat
Figure Examples of 256 x 256 images generated by I-rectified flow.
(A) Cat - Wild Animals (B) Wild Animals - Cat
(C) MetFace - CelehA Face
(D) CelebA Face + MetFace
Figure 8: Samples of 1-rectified flow simulated with N = 100 Fuler steps hetween different domains.
As the main goal here to obtain good visual results, we are not interested in faithfully transferring Xu Wo to X1 that exactly follows TT. Rather, we interested in transferring the image styles while preserving the identity of the main object in the image. For example, when transferring hu- man face image cat face, are interested in getting unrealistic face of human-cat hybrid that still "looks like" the original human face. To achieve this, let ((x)) be a feature mapping of image w representing the styles that we are interested in transferring. Let X2 = tX1 + -) He h(X,) follows an ODE of dH, = V)(X,) (X1 - df. Hence, to ensure that the style is transferred correctly, we propose learn v such that H, h( (Z)) with dZ, v v(Z,, t)dt approximates H, as much as possible. Because dH, Vh(Z.) Te (z, t)dt, we minimize the loss:
min ( E Vh(x,) (X, - Xo v( 8))
27
dt,
t)
(4)
In practice we set h()) to be latent representation of classifier trained to distinguish the images from the two domains To, #1, fine-tuned from pre-trained ImageNet (Tan & Le, 2019) model. Intuitively, V (zh(z) serves as saliency score and re-weights coordinates so that the loss in (4)
focuses on penalizing the error that causes significant changes on
I-Reeztifed Flase
2-Rlaciffod Flowr I-Routified 2-Ractifiod I-Raztified 2-Rectified
LI0
N N
EIO
Figure 9: Samples results of and 2-rectified flow simulated with and 100 Euler steps. Experiment settings We set the domains To, गı to be pairs of the AFHQ (Choi et al., 2020), MetFace (Karras et al., 2020) and CelebA-HQ (Karras al., 2018) dataset. The results are shown by initializing the trained flows from the test data. The training and network configurations follow Section 3.1. See Appendix E for details.
Results Figure 1, 8, 9 show examples of results of 1- and 2-rectified flow simulated with Euler that rectified flows successfully transfer
method with different number of steps N. We the styles and generate high quality images. Moreover, 2-rectified flow returns good results with a single Euler step. See more examples in Appendix E.
4 CONCLUSIONS
Rectified flow provides simple and clean framework for learning transport mappings from data: (1) it can be applied to both generative and transfer modeling; (2) is able to learn fast ODEs (even one- step) by learning straight flows; (3) it provides new way for understanding the diffusion models and their ODE variants; (4) is purely based, avoiding SDE models both conceptually and algorithmically; (5) the theoretical and algorithmic insights to optimal transport of independent interest; (6) the idea of causalizing interpolation processes provides general framework related to transport mapping problems and is amenable rigorous theoretical analysis.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research is supported by NSF CAREERI 1846421, SenSE2037267, EAGER-2041327, Office of Navy Research, and NSF AI Institute for Foundations of Machine Learning (IFML).
REFERENCES
Michael S Albergo and Eric Vanden Eijnden. Building normalizing flows with stochastic inter- polants. arXiv preprint (xXi::2209./5577/ 2022.
Luigi Ambrosio and Gianluca Crippa. Existence, uniqueness, stability and differentiability prop- erties of the flow associated to weakly differentiable vector fields. In Transport equations and multi-D hyperbolic conservation laws, pp. 3-57, Springer, 2008
Luigi Ambrosio, Elia Brué, and Daniele Semola. Lectures on optimal transport. Springer, 2021.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3)311-332, 1982.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 214-223. PMLR, 2017.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic- :PM:: analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint Stti::2001.0503.. 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Tianrong Chen, Guan- Houn-Horng Liu, and Evangelos Theodorou. Likelihood training Schrödinger sdes theory. arXiv preprint arXiv:2 /2211 2021.
bridge using
Jooyoung Choi. Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprint 1xxi::200.002::: 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 8188-8197, 2020.
Max Daniels, Tyler Maunu, and Paul Hand. Score-based generative neural networks for large-scale optimal transport. Advances in neural information processing systems, 34:1 19955-12295, 2021.
Valentin De Bortoli, James Thomton, Jeremy Heng, and Arnaud Doucet. Diffusion Schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. Ad- vances Neural Information Processing Systems, 34, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516. 2014.
Laurent Dinh, Jascha Sohk-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0880 2016.
Alessio Figalli and Federico Glaudo. An Invitation to Optimal Transport, Wasserstein Distances, and Gradient Flows. 2021.
R Flamary, N Courty, D Tuia, and A Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell, 1, 2016.
Hans Föllmer. An entropy approach to the time reversal of diffusion processes. Stochastic Differential Systems Filtering and Control, pp. 156-163. Springer, 1985.
10

---

Published as a conference paper at ICLR 2023
Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filip- pone, and Pietro Michiardi. How much is enough? study on diffusion times in score-based generative models. arXiv preprint arXIv:2206.05173, 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprini arXiv:2007.01434, 2020.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. arXiv preprint arXiv:2205. 11495, 2022.
Tatsunori Hashimoto, David Gifford, and Tommi Jaakkola. Learning population-Ieve level diffusions with generative mns. In International Conference on Machine Learning, pp. 2417-2426. PMLR, 2016.
Ulrich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability, pp. 1188-1205, 1986.
Eric Heitz, Laurent Belcour, and Thomas Chambon. Iterative $ (laaphaaS-dde)blendding Learning deterministic mapping between arbitrary densities, 2023. httpsts:: htt.tpppss//peeeeevvee... net/
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:220..03458 2022.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1134, 2017.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. TransGAN: Two pure transformers make one strong GAN, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:1 33:12104-1211 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint urXiv:2206.00366 2022.
Valentin Khrulkov and Ivan Oseledets. Understanding DDPM latent codes through optimal trans- port. arXiv preprint arXiv:2202.07477 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412v.1412.6980, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv:1312.61 2013.
arXiv preprint
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2023
Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, Alexander Filippov, and Evgeny Burnaev. Do neural optimal transport solvers work? a continuous wasserstein-2 benchmark. Advances in Neural Information Processing Systems, 34:14593-14000 14699, 2021.
Alexander Korotin, Daniil Selikhanovych, and Evgeny Burneev. Neural optimal transport. arXiv preprint arXiv:2201.12220 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Thomas G Kurtz. Equivalence of stochastic equations and martingale problems. In Stochastic analysis 2010, pp. 113-130. Springer, 2011.
Tuomas Kynkäääniemii Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Hugo Lavenant and Filippo Santambrogio. The flow map of the tokker-planck equation does not provide optimal transport. Applied Mathematics Letters, pp. 108225, 2022.
Hugo Lavenant, Stephen Zhang, Young-Heon Kim, and Geoffrey Schiebinger. Towards a mathe- matical theory of trajectory inference. arXiv preprint arXiv:2102.09204 2021.
Junhyeok Lee and Seungu Han. Nu-wave: A diffusion probabilistic model for neural audio upsam-
pling. arXiv preprint arXiv:2104.02322 2021.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint urXiv:2205.142177 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 1xXi::2200.02747, 2022. 2022.
Qiang Liu. Rectified flow: marginal preserving approach to optimal transport. arXiv preprint 12777 2022a.
Qiang Liu. On rectified flow and optimal coupling. preprint, 2022b.
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fuse- dream: Training- frining-free free text-to-image -maattt-imeegeeee generation with improved clip+ gan space optimization. arXiv 2021.
preprint 1rXiv:2112.01575
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208. 14699, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint TxXiv:2206.00927, 2022.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2 02388, 2021.
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205. /2524, 2022.
Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning. pp. 6672- 6681. PMLR, 2020.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
12

---

Published as a conference paper at ICLR 2023
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint drXiv:2103.16091, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
Kirill Neklyudov, Daniel Severo, and Alireza Makhzani. Action matching: A variational method for learning stochastic dynamics from samples. arXiv preprint arXiv:22 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv: /2. 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 3162-81711 PMLR, 2021.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate continu- ous normalizing flows via optimal transport. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 2223-9232, 2021.
George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res., 22(57):1-64 2021.
Stefano Peluchetti. Non-denoising forward-time diffusions. 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1411 1415, 2019.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 1165-6):35-6077 2019.
Vadim Popov, Ivan Vovk. Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599 8608. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 2022.
Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. Interna- tional conference machine learning, pp. 1530-1538. PMLR, 2015.
Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. arXiv preprint arXiv:2110.02999, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 1487, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(8-633):94 2015.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pp. 1-10, 2022.
Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathicu Blondel. Large-scale optimal transport and mapping estimation. arXiv preprint 171 171
2017.
13

---

Published as a conference paper at ICLR 2023
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2C: Diffusion-decodinn mod- els for few-shot conditional generation. Advances in Neural Information Processing Systems, 34: 12533-12548, 2021.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for applications, volume 11006, pp- 369-386, SPIE, 2019.
operations
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training coore-based generative models. 2020.
Advances in neural information processing systems, 33:
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training score- based diffusion models. Advances in Neural Information Processing Systems, 34, 2021.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for taage-to-image translation. arXiv preprint arX1v:2203.08882, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443 450. Springer, 2016.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Anastasiya Tanana. Comparison of transport map generated by heat flow interpolation and the optimal transport brenier map. Communications in Contemporary Mathematics, 23(06))2550025, 2021.
Giulio Trigila and Esteban G Tabak. Data-driven optimal transport. Communications on Pure and Applied Mathematics, 69(4) 648, 2016.
Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In Conference on Learning Theory, pp. 3084-3114, 14. PMLR, 2019.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. Advances in Neural Information Processing Systems, 34: 1287-1180222 2021.
Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving Schrödinger bridges via maximum likelihood. Entropy, 23(9): 134, 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5018-5027, 2017.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 1038/0//411 592-019--666--22
14

---

Published as a conference paper at ICLR 2023
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via Schrödinger bridge. In Interaational Conference on Machine Learning, pp. 10794- PMLR, 2021.
Rose E Wang, Esin Durmus, Noah Goodman, and Tatsunori Hashimoto. Language modeling via stochastic processes. arXiv preprini 1rXiv:2203.113700 2022.
Antoine Wehenkel and Gilles Louppe. Diffusion priors in variational autoencoders. arXiv preprint arXiv:2106. 1567,,, 2021.
Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion-based molecule generation with informative prior bridges. arXiv preprint, 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. arXiv preprint arXiv:2112.07804, 2021.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203. 09481, 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv 2015.
preprint
Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. arXiv preprint arXiv:2209. 02606, 2022a.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 00044.339900 13902, 2022.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gDDIM: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05564, 2022b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. EGSDE: Unpaired image-to-image translation via energy-guided stochastic differential equations. arXiv preprini urXiv:2207.0663 2022.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. Advances in Neural Information Processing S 0147559-77700 2020.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion proba- bilistic models. arXiv preprint 200,002002..067.,, 2022.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
15

---

Published as a conference paper at ICLR 2023
Roadmap
The appendix is structured follows:
Appendix A discusses related works on generative models, neural ODE, diffusion models, and
optimal transport.
Appendix B discusses the form and estimation of velocity field x of rectified flow.
Appendix C introduuess nonlinear extension of our method, which include probability flow ODEs
(Song et al., 2020b) and DDIM (Song et al., 2020a) as special cases,
Appendix D presents the full course theoretical analysis of rectified flow.
Appendix E gives the details of the experiments and additional experiment results.
A RELATED WORKS AND DISCUSSION
Learning one-step models GANs (Goodfellow et al.. 2014; Arjovsky et al., 2017: Liu et al., 2021), VABs (Kingma & Welling, 2013), and (discrete-time) normalizing flows (Rezende & Mo- hamed, 2015; Dinh et al., 2014; 2016) have been three classical approaches for learning deep gen- erative models. GANs have been most successful in terms of generation qualities (for images in particular), but suffer from the notorious training instability and mode collapse issues due to use of minimax updates. VAEs and normalizing flows are both trained based on the principle of maximum likelihood estimation (MLE) and need to introduce constraints the model architecture and/or special approximation techniques to ensure tractable likelihood computation: VAEs typically use conditional Gaussian distribution in addition to the variational approximation of the likelihood; normalizing flows require to use specially designed invertible architectures and need to copy with calculating expensive Jacobian matrices.
The reflow-distillttion approach in this work provides another promising approach to training one- step models, avoiding the minimax issues of GANs and the intractability issues of the likelihood- based methods.
Learning ODEs: MLE and PF-ODEs There are two major upproaches for learning neural ODEs: the PF-ODEs/DDIM approach discussed in Section C, and the more classical MLE based approach of Chen et al. (2018).
The MLE approach. In Chen et al. (2018), neural ODEs are trained for learning generative models by maximizing the likelihood of the distribution of the ODE outcome Z1 time t = under the data distribution T. Specifically, with observations from Ii it estimates neural drift v of an ODE dZ. = o( Ze, l)dt by
min D(ii p po))))
where (T: :) denotes KL divergence (or other discrepancy measures), and pa. is the density of Z1 following dZ, = v(z,, t)dt from Zu ~ To; the density of To should be known and tractable to calculate.
By using an instantaneous change of variables formula, it was observed in Chen et al. (2018) that the likelihood of neural ODEs easier compute than the discrete-time normalizing flow without constraints on the model structures. However, this MLE approach is still computationally expensive for large scale models it requires repeated simulation of the ODE during each training step. In addition, as the optimization procedure of MLE requires to backpropagate through time, it can easily suffer the gradient vanishing/exploding problem unless proper regularization is added.
Another fundamental problem is that the MLE (5) of neural ODEs is theoretically under-specified, because MLE only concerns matching the law of the final outcome Z, with the data distribution Ni, and there are infinitely many ODEs to achieve the same output law of Zi while traveling through different paths. A number of works have been proposed remedy this by adding regularization terms, such these based transport costs, to favor shorter paths; (Nichol & Dhariwal, 2021; Onken et al., 2021). Without a regularization term, the ODE learned by MLE would be implicitly determined by the initialization and other parammerreeters of the optimizer used to solve (5).
Probability Flow ODEs. The method of PF-ODEs (Song et al., 2020b) and DDIM (Song et al., 2020a) provides different approach to learning ODEs that avoids the main disadvantages of the MLE approach, including the expensive likelihood calculation, training- time simulation of the ODE models, and the need of backpropagation through time. However, because PF-ODEs and DDIM
16

---

Published as a conference paper at ICLR 2023
were derived as the side product of learning the mathematically more involved diffusion/SDE mod- els, their theories and algorithm forms were made unnecessarily restrictive and complicated. The nonlinear rectified flow framework shows that the learning of ODEs can be approached directly in a very simple way, allowing us to identify the canonical case of linear rectified flow and open the door of further improvements with flexible and decoupled choices of the interpolation curves X and initial distributions To-
Viewed through the general non-linear rectified flow framework, the computational and theoretical drawbacks of MLE can be avoided because we can simply pre-determines the "roads" that the ODEs should travel through by specifying the interpolation curve X,, rather than leaving it for the algo- rithm to figure out implicitly. It is theoretically valid to pre-specify any interpolation X, because the neural ODE highly werr-parameterize as a generative model: when e is a universal approximator and To is absolutely continuous, the distribution of Zi can approximate any distribution given any fixed interpolation curve X,. The idea of rectified flow is to the simplest geodesic paths for X,.
Learning SDEs with denoising diffusion Although the scope of this work is limited to learning ODEs, the score-based generative models (Song & Ermon, 2019; 2020; Song et al., 2020b; 2021) and denoising diffusion probability models (DDPM) (Ho et al., 2020) are of high relevance as the basis of PF-ODEs and DDIM. The diffusion/SDE models trained with these methods have been found outperforming GANS in image synthesis in both quality and diversity (Dhariwal Nichol, 2021). Notably, thanks to the stable and scalable optimization-basee training procedure, the diffu- sion models have successfully used in huge text-to-image generation models with astonishing results (e.g., Nichol et al., 2021; Ramesh et al., 2022: Saharia et al., 2022). It has been quickly popularized in other domains, such as video (e.g.. Ho et al.. 2022; Yang et al.. 2022; Harvey et al., 2022), music (Mittal et al., 2021), audio (e.g., Kong al., 2020; Lee & Han, 2021; Popov et al., 2021). and text (Li et al., 2022; Wang et al.. 2022), and more tasks such as image editing (Zhao et al., 2022; Meng et al., 2021; Zhang et al., 2022a). A growing literature has been developed for improving the inference speed of denoising diffusion models, an example of which is the PF-ODEs/DDIM approach which gains speedup by turning SDEs into ODEs. We provide below some examples of recent works, which is by no mean exhaustive.
Improved training and inference. A line of works focus on improving the inference and sampling procedure of denoising diffusion models. For example. Nichol & Dhariwal (2021) presents a few simple modifications of DDPM improve the likelihood, sampling speed, and generation quality. Karras et al. (2022) systematic exams the design space of diffusion generative models with empirical studies and identifies number of training and inference recipes for better generative quality with fewer sampling steps. Zhang & Chen (2022) proposes a diffusion exponential integrator sampler for fast sampling of diffusion models. Lu et al. (2022) provides a customized high order solver for PF-ODEs. Bao et al. (2022) provides analytic estimate of the optimal diffusion coefficient.
Combination with other methods. Another direction is to speed up diffusion models by combining them with GANs and other generative models. DDPM Distillation (Luhman & Luhman, 2021) accelerates the inference speed by distilling the trajectories of a diffusion model into a series of conditional GANs. The truncated diffusion probabilistic model (TDPM) of (Zheng al., 2022) trains a GAN model as To so that the diffusion process can be truncated to improve the speed; the similar idea was explored in Lyu et al. (2022); Franzese et al. (2022), and (Franzese et al., 2022) provides an analysis on the optimal truncation time. (Sinha et al., 2021; Wehenkel & Louppe, 2021; Vahdat al., 2021) learns a denoising diffusion model in the latent spaces and combines it with variational auto encoderss..ss These methods can be potentially applied to rectified flow to gain similar speedups for learning neural ODEs.
Unpaired Image-to- -m-ge translation. The standard denoising diffusion and PF-ODEs methods focus on the generative task of transferring a Gaussian noise (mx) to the data (TI). A number of works have been proposed to adapt it to transferring data between arbitrary pairs of source-target domains. For example, SDEdit Meng et al. (2021) synthesizes realistic images guided by an input image by first adding noising to the input and then denoising the resulting image through pre-trained SDE model. Choi et al. (2021) proposes a method to guide the generative process of DDPM to generate realistic images based on given reference image. Su et al. (2022) leverages two PF-ODEs for image translation, one translating source images to a latent variable, and the other constructing the target images from the latent variable. Zhao et al. (2022) proposes energy-guided approach
17

---

Published as a conference paper at ICLR 2023
that employs an energy function pre- vre-trained on the source and target domains to guide the inference process of a pretrained SDE for better image translation. In comparison, our framework shows that domain transfer can be achieved by essentially the same algorithm as generative modeling, by simply setting To to be the source domain.
Diffusion bridges. Some recent works Peluchetti (2021); Liu al. (2022) show that the design space of denoising diffusion models can be made highly flexible with the assistant of diffusion bridge processes that are pinned to fixed data point at the end time. This reduces the design of denoising diffusion methods to constructing a proper bridge processes. The bridges in Song et al. (2020b) are constructed by a time-reversal technique, which can be equivalently achieved by Doob's h- transform as shown in Peluchetti (2021); ); Liu et al. (2022), and more general construction techniques are discussed in Liu et al. (2022); Wu et al. (2022). Despite the significantly extended design spaces, an unanswered question is what type of diffusion bridge processes should he preferred. This question made challenging because the presence of diffusion noise and the need of advanced stochastic calculus tools make it hard to intuit how the methods work. By removing the diffusion noise, our work makes it clear that straight paths should be preferred. We expect that the idea can be extended to provide guidance designing optimal bridge processes for learning SDEs.
Schrodinger bridges. Another body of works (Wang et al., 2021; De Bortoli et al., 2021; Chen et al., 2021; Vargas et al., 2021) leverages Schrodinger bridges (SB) as an alternative approach to learning diffusion generative models. These approaches are attractive theoretically, but casts significant computational challenges for solving the Schrodinger bridge problem.
Re-thinking the role of diffusion noise The introduction of diffusion noise was consider essential due to the key role it plays in the derivations of the successful methods (Song et al., 2020b; Ho al., 2020). However, as rectified flow can achieve better or comparable results with a ODE-only framework, the role of diffusion mechanisms should be re-examed and clearly decoupled from the other merits of denoising diffusion models. The success of the denoising diffusion models may be mainly attributed to the simple and stable aptimizationn-aasee training procedure that allows to avoid the instability issues and the need of case-by-case tuning of GANs, rather than the presence of diffusion noises.
Because our work shows that there is no need to invoke SDE tools if the goal is to learn ODEs, the remaining question is whether we should learn an ODE an SDE for given problem. As already argued by a number of works (Song et al., 2020b;a; Karras et al., 2022), ODEs should be preferred over SDEs in general. Below is detailed comparison between ODEs and SDEs.
Conceptual simplicity and numerical speed. SDEs are more mathematically involved and are more difficult understand. Numerical simulation of ODEs simpler and faster than SDEs.
Time reversibility. It is equally easy to solve the ODEs forwardly and backwardly. In comparison, the time reversal of SDEs (c.g., Anderson, 1982; Haussmann & Pardoux, 1986; Föllmer, 1985) is more involved theoretically and may not be computationally tractable.
Latent spaces. The couplings (Zo, Z1) of ODEs are deterministic and yield low transport cost in the case of rectified flows, hence providing good latent space for representing and manipulating outputs. Introducing diffusion noises make (Zo, Z1) more stochastic and hence less useful. In fact, the (7o 71) given by DDPM Ho et al. (2020) and the SDEs of Song et al. (2020b) and hence useless for latent presentation.
Training difficulty. There is no reason to believe that training an ODE is harder, if not easier, than training an SDE sharing the same marginal laws: the training loss of both cases would share the distributions of covariant and differ only on the targets. In the setting of (Song et al., 2020b), the loss functions are equivalent up to a linear reparameterization.
Expressive power As every SDE can be converted into ODE that has the same marginal distribution using the techniques in Song et al. (2020a;b) (see also Villani (2009)), ODEs are as powerful SDEs for representing marginal distributions, which is what needed for the transport mapping problems considered in this work. On the other hand, SDEs may be preferred if we need to capture richer time-correlation structures.
Manifold data When equipped with neural network drifts, the outputs of ODEs tend to fall into a smooth low dimensional manifold, key inductive for structured data in AI such images and text.
18

---

Published as a conference paper at ICLR 2023
In comparison, when using SDEs to model manifold data, one has carefully anneal the diffusion noise to obtain smooth outcomes, which causes slow computation and a burden of hyperparameter tuning. SDEs might be more useful in for modeling highly noisy data in areas like finance and economics, and areas that involve diffusion processes physically, such as molecule simulation.
Optimal vs. straight transport Optimal transport has been extensively explored in machine learning as a powerful way to compare and transfer between probability measures. For the transport mapping problem considered in this work, natural approach is finding the optimal coupling (Zu. Z1) that minimizes a transport cost Eie(Z] Zo}] for given c. The most common choice of is the quadratic cost c(-) = 11-11-.
However, finding the optimal couplings, especially for high dimensional continuous measures, is highly challenging computationally and is the subject of active research; for example (Seguy et al., 2017; Korotin et al., 2021; 2022; Makkuva al., 2020; Rout et al., 2021; Daniels et al., 2021). In addition, although the optimal couplings are known to have nice smoothness and other regularity properties, not necessary to accurately find the optimal coupling because the transport cost do not exactly align with the learning performance of individual problems; e.g., Korotin et al. (2021).
In comparison, reflow procedure finds straight coupling, which is not optimal w.r.t given (see Section D.4). From the perspective of fast inference, all straight couplings are equally good because they all yield straight rectified flows and hence be simulated with Euler step.
Algorithm 2 Train (Data)
Input: Data=(x0, x1}
Output:
Model u(z, t) for the rectified flow
initialize Model
for x0, x1 in Data: * x0, x1: samples from Tor 3
Optimizer.zero.grae
t = torch. rand (batchsize)
Randomly sample LE [0,1]
Loss ( Model (t+x1+ (1-t) +x0, t) (x1-x0) ) pow (2) .maan ()
baskwarck))))
Optimizer. step( ()
return Model
Algorithm Sample (Model, Data)
Input: Model v(x, t) of the rectified flow
Output: draws of the rectified coupling (Zo, Zi)
coupling = []
for x0, in Data: x0: samples from To (batchsizexdimm
xl = model. ODE solver (x0)
coupling aoupling.append (x0, x1))
return coupling
Algorithm Reflow (Data)
Input: Data=(x0, x1}
Output: draws of the K-th rectified coupling Coupling Data
for k = l,..., K:
Model Train (Coupling)
Coupling sample (Model, Data) return Coupling
19

---

Published as a conference paper at ICLR 2023
==1 =
N =
N=1 =
=
VE ODE
I-Rectified Flow
2-Rectified Flow
sub-VP ODE
Figure 10: Few-step generation with different ODEs. Compared with VE,VPsub-VP ODE, 1-rectified flow generate blurry images using only 1,2,3 steps. After one time of rectification, 2-rectified flow can generate clear images with 1,2,3 steps.
B ON THE VELOCITY FIELD xX OF RECTIFIED FLOW
On the velocity field xx If Xo yields a conditional density function p(2o x1) when conditioned
on X1 = 21, then the optimal velocity field v (z, = B[X1 Xu X+
z] can be represented by
1 X1 ne m(x)==) (x(1
1X1
LX1
xx (z,
X1
E
X1
where the expectation E [:] is taken w.r.t. X1 Ti. This can be seen by noting that Xo and X1 - Xn = - when conditioned on X, = 2. Hence, if p is positive and continuous everywhere, then x is well defined and continuous on R x [0, 1). Further, if log ne is continuously differentiable w.r.t. 2, we can show that
8-tX1
(z, T [((X) - 2) (o - (e (X1,
Note that dZ, = x (Zz, d) is guaranteed to have a unique solution if x is uniformly Lipschitz continuous on [0, for any 1.
If Xo = does not yield a conditional density function, (z, t) may be undefined or dis- continuous, making the ODE dZ, = x (Z,, t)dt ill-behaved. simple fix is add X, with Gaussian noise Ç ~ (O o² T) independent of (Xo, X1) to yield smoothed variable X, Xo and transfer Xo to X, using rectified flow. This would effectively give randomized mapping of form T(X, + {) transporting No to ni.
Smooth function approximation Following (6), we can exactly calculate x if the conditional density function p(-1æ11) exists and is known, and na is the empirical measure of a finite number of points (whose expectation can be evaluated exactly). In this case, running the rectified flow forwardly would precisely recover the points in æ1. This, however, is not practically useful in most cases as it completely overfits the data. Hence, it is both necessary and beneficial to fit x with a smooth function approximator such as neural network or non non-parametric models, obtain smoothed distributions with novel samples that are practically useful.
Deep neural networks are doubt the best function approximators for large scale problems. For low dimensional problems, the following simple Aidarrya-Watson style parameerie estimator of x can yield a good approximation to the exact rectified flow without knowing the conditional density
Xi - (Xt, (2, { - [
X.
[x,2]X and Kn (x, 2) is a smoothing kernel with bandwith param- eter h 0 that measures the similarity between and Taking the Gaussian RBF kernel Nn (I,, 2) exp( || =11² /2h2), then when h 0+, can be shown that (,,,(, converges points that can be attained by X, (i.e., the conditional expec- On points that X, can attain, X. (2, extrapolates the value
where wn x), = nn (x.2))))
to x (2,t)= E
X1=2 X,
tation E. X, exists.
20

---

Published as a conference paper at ICLR 2023
by finding the X, that is close to In practice, we replace the expectations in (7) with empirical averaging. We find that v. performs well in practice because it is a mixture of linear functions that always point to point in the support of TT
C A NONLINEAR EXTENSION OF RECTIFIED FLOW
We present nonlinear extension of rectified flow in which the linear interpolation X2 is replaced by any dime-iiffeeentialee curve connecting Xo and X1. Such generalized rectified flows can still transport To to (Theorem D.3), but no longer guarantee to decrease convex transport costs, or have the straightening effect. Importantly, the method of probability flows (Song et al., 2020b) and DDIM (Song et al., 2020a) can be viewed (approximately) as special cases of this framework, allows us to clarify the connection with and the advantages over these methods.
Let X = (Xt: / E [0, 1|} be any time-differentiable random process that connects Xo and X1. Let X, be the time derivative of X,, More specifically, what is standard in stochastic process, the stochastic process X, can be viewed as a measurable function X, = ((t,w), where w can be viewed as a some "random seed". The time-derivative is X. = 0 (1,
The (nonlinear) rectified flow induced from X is defined as dZz x (Ze, t)dt,
with Zo Xu: and xx (2. =E E
X1 X1 =
We can estimate x by solving
v(x,, t) - x'o1"] dt,
min
E We
(8)
where w2: (0, - (0, +00) is positive weighting sequence (w, 1 by default). When using the linear interpolation X, = (X1 + (1 X). we have X, X1 Xo and (8) with w. reduces to (1). As we show in Theorem D.3, the flow Z given by this method still preserves the marginal laws of X, that is, Law( Z,)
Law(X,) Vt E [0, 1]. and hence (Zo. Z1) remains to be coupling of To. IT However, if X is not straight, (Zo. 71) no longer guarantees to decrease the convex transport costs over (Xo, X1). More importantly, the reflow procedure no longer straightens the paths of Z,.
A simple class of interpolation processes is X, = a, + 3.Xo where a and B, are two differ- entiable sequences that satisfy αn = Bn I and ag B, 0 to ensure that the process equals X1 at the starting and end points. In this case, have X. å; + 3. in (8) where á; and 3r are the time derivatives of an and 3, The shape of the curve is controlled by the relation of an and Sr. If we take 3r = 1 - a for all f, then X, have straight paths but does not travel at a constant speed; it can be viewed as time changed variant of the canonical case X, tXx + (1 - X) when t is reparameterized to a; When 3, # 1 a₁ the paths of X, are not straight lines except some special cases (e.g., dg X1 = or X, = 0, or X = ax1 for some a E R).
C.I PROBABILITY FLOW ODES AND DDIM
The probability flow ODEs (PF-ODEs) Song al. (2020b) and denoising diffusion implicit models (DDIM) Song et al. (2020a) are methods for learning ODE-based generative models of π from a spherical Gaussian initial distribution To, derived by converting a SDE learned by denoising dif- fusion methods to an ODE with equivalent marginal laws. In Song et al. (2020b), three types of PF-ODEs are derived from three types of SDEs learned as score-based generative models, including variance-exploding (VE) SDE, variance- preeerrvig (VP) SDE, and sub- VP SDE, which we denote by VE ODE, VP ODE, and sub-VP ODE, respectively. VP ODE is equivalent to the continuous time limit of DDIM, which is derived from the denoising diffusion probability model (DDPM) Ho et al. (2020). As the derivations of PF-ODEs and DDIM require advanced tools in stochastic calculus, we limit our discussion the final algorithmic procedures suggested in Song et al. (2020b); Ho et al. (2020). The readers are referred to Song et al. (2020b;a) for the details.
Proposition C.1. All variants of PF-ODEs can be viewed as instances of (8) when using Xt a,X, + Be for some ae. 3r with 0q = 1, = O, where & ~ NOO0,) is a standard Gaussian random variable.
21

---

Published as a conference paper at ICLR 2023
Here need to use introduce to replace Xo because the choices of a; and Be suggested in Song et al. (2020b;a); Ho et al. (2020) do not satisfy the boundary condition of ag = and Bo = 1 at t = 0, and hence Xo # Instead. in these methods, the initial distribution X, To is implicitly defined as Xo = Og 1 + Anc, which is approximated by Xo s gog by making X1 < 805-- Hence, To is set be N(0 36 [) in these methods. Viewed through our framework, there is no reason to restrict to be N (0, P0 I). or not set do = 0, 80 = to avoid the approximation.
Rectified flow (ar = I, Br = 1)
VP ODE (or in (9), 8:
sub-VP ODE (as in (9), 8 = ai)
V1- )
1-Rectified Flow
Rectified
I-Rectified
2. 2-Rectified Flow
1-Roctified Flow
2-Roctified
Figure 11: Comparing rectified flow with VP ODE and sub-VP ODE when To N/ 7) (purple dots) and
is low variance Gaussian mixture shown the red dots. The linear rectified flow yields nearly straight trajectories with one step of reflow. But the trajectories of VP ODE and sub-VP ODE curved and be straightened by reflowing.
Time- Dime-Diicretization Steps
Rectified Flow
VP ODE
sub-VP ODE (9), &
VP ODE (const speed)
O2 8
az & V1 of
f
N
==
=
= 100
Figure 12: Trajectories of different methods when varying the number of discrctization steps N (purple dots: no; red dots: wi; orangle dots: intermediate steps; blue curves: flow trajectories). The rectified flow travels straight lines and progresses uniformly time; generates Euler step, and quickly covers the whole distribution тı with more steps (in this case N = 2 is sufficient). In comparison, VP ODE and sub-VP VP ODE travel in curves with non-uniform speed: they tend to be slow in the beginning and speed up in the later phase (much of the update happens when t20.5). The non-uniform speed can avoided by setting au (see last column).
mean क when simulated with a single
VP ODE and sub- VP ODE lowing shared ay:
The VP ODE and sub-VP ODE of Song et al. (2020b) use the fol-
1 (sub-)VP ODE: 04 = exp ( 1)2
1 1u1 (((1-))
n):
default values: a 19.9, 0.1,
where the default values of a, b are chosen to match the continuous time limit of the shared training procedure of DDIM and DDPM. The difference VP ODE and sub-Vi VP ODE is on the choice of Be, given as follows:
ODE 3e = V a af.
sub- VP ODE: B 1 af
(10)
As 3n 1 in both VP and sub-VP ODE, the mo in both cases are taken /O I).
The choices of at, 3r above are the consequence of the SDE-based derivation in Song et al. (2020b).
However, they not well-motivated when exam the path properties of the induced ODEs:
22

---

Published as a conference paper at ICLR 2023
Non-straight paths: Due choices of Br in (10), the trajectories of VP ODE and sub- VP ODE are curved in general, and can not be straightened by the reflow procedure. We should choose 3 a, induce straight paths.
Non-uniform speed: The exponential form of a in (9) consequence of using Orastein-Uhlenbere processes in the derivation of SDE models (Song et al., 2020b; Ho al., 2020). However, there no clear advantage of us- ing (9) for ODEs. As shown in Figure 12, the On and B, of VP and sub- VP ODE change slowly the early phase (tg0.5). As a result, the flow also moves slowly in begin- ning and hence most of the updates are concentrated in the later phase. Such non-uniform update speed, in addi- tion the non-straight paths, make VP ODE and sub- VP ODE perform sub-optimally when using large step sizes, even for transport between simple spherical Gaussian distributions (see Figure 12). As we show in the last column of Figure 12, changing the exponential a to the linear function a; = t in VP ODE allows us to get a uniform update speed while preserving the same continuous time trajectories.
d, Rectified
Rectified
VP ODE
A. VP ODE
0.5
op sub-VP ODE
sub-VP
0.0
0.5
15
Figure 13: t vs. du., , of different methods.
VE ODE The VE ODE of Song et al. (2020b) uses or = 1 and Bg min Vr where min 0.01 by default r is set such that mmax == m is as large as the maximum Euclidean distance between all pairs of training data points from Ti (Technique 1 of Song & Ermon (2020)). Assume that o maa is much larger than both a2 min and the variance of X1, then X,- X1 + 8og R Tmaxx, and we can set the initial distribution to be ~ _N(,, o omax /). which has much larger variance than TT. Hence, VE ODE can not be applied to (and not shown in) the toys in Figure 11 and Figure 12. As the case of (S ODE, the restriction that max is unnatural viewed from our framework. On the other hand, the trajectories of X, in VE ODE are indeed straight lines, because the direction of X, = B& is always the same as &. However, the choice of B. causes non-uniform speed issue similar to that of (sub-)VP ODE.
is in fact unnecessary and requirement
Following Song et al. (2020b); Ho et al. (2020), a line works have been proposed to improve the choices of at, 3e, but remain to be constrained by the basic design space from the SDE-to-ODE derivation; see for example Nichol & Dhariwal (2021); Karras et al. (2022); Zhang et al. (2022b).
To summarize, the simple nonlinear rectified flow framework in (8) both simplifies and extends the existing framework, and sheds a number of importance insights:
Learning ODEs be considered directly and independently without resorting to diffusion/SID methods;
The paths of the learned ODEs can be specified by any smooth interpolation curve X, of Xo and X1;
The initial distribution =o can be chosen arbitrarily, independent with the choice of the interpola- tion X,-
The canonical linear interpolation X, tx, + (1 X)x, should be recommended default choice.
On the other hand, non-linear choices of X, can be useful when we want incorporate certain non- Euclidan geometry structure of the variable, or want to place certain constraints on the trajectories of the ODEs. We leave this for future works.
D THEORETICAL ANALYSIS
We present the theoretical analysis for rectified flow. The results are summarized follows.
[Section D.1] All nonlinear rectified flows with any interpolation X, preserve the marginal laws.
[Section D.2] The rectified flow (with the canonical linear interpolation) reduces convex transport costs.
Section D.3] Reflow guarantees to straighten the (linear) rectified flows.
23

---

Published as a conference paper at ICLR 2023
[Section D.4] We clarify the relation between straight couplings and c-optimal couplings.
D.1 THE MARGINAL PRESERVING PROPERTY
The marginal preserving property that Law (Z+) Law for VT is a general property of the nonlinear rectified flows in (8), regardless whether the interpolation X, is straight not.
Definition D.1. For path-wise continuously differentiable random process X its expected velocity xx xxx defined as
{Xt: e [O,
xx (x, = B X₁ =x]: Vo E supp(X.).
For x & supp(X.). the conditional expectation is not defined and we set xx arbitrarily, say x (x, 1)
Definition D.2. We call that X is rectifiable v is locally bounded and the solution of the integral equation below exists and is unique:
Zz To + [ xx (Z,,t)), Vt E [0, Zo Xo-
(11)
In this case, Z { E [0, 1]} is called the rectified flow induced from X. Theorem D.3. Assume X is rectifiable and Z is its rectified flow. Then Law (Z+) = (X,( for Vt [0, 1].
Proof. For any compactly supported continuously differentiable test function h: Rd R, have
d [ E(V)(X,) Tx, V(())) (Xe:t)), (Xx:(ttt)) dt
(12)
where we used x tX, = E[X, X By definition, this is equivalent to that Ti == Law(X,) solves in the sense of distributions the continuity equation with drift vz == x (,,t):
क + V
(n TT) = O.
(13)
To see the equivalence of (12) and (13), we can multiply (13) with h and integrate both sides:
= f h(ftz
x mz)) = f hofre Vh
d B[N(x.) E|VA(X,) xx (X,, t)],
dt
where we use integration by parts that f JAV (v n.) - Vh (uxTTT). m.).
Because Z, is driven by the same velocity field xx its marginal law Law(Z;) solves the very same equation with the same initial condition (7o Xo). Hence, the equivalence of Law and Law(X) (2) follows the solution of (13) is unique, which is equivalent to the uniqueness of the solution of dZ, x.X (Z,, t) following Corollary 1.3 of Kurtz (2011) (see also Theorem 4.1 of Ambrosio & Crippa (2008)).
D.2 REDUCING CONVEX TRANSPORT COSTS
The fact that (Zo, Z1) yields no larger convex transport costs than Xi) is consequence of using the special linear interpolation X, = 1X1 + (1-1)Xo as the geodesic of Euclidean space.
Definition D.4. A coupling (Xa, X1) is called rectifiable if its linear interpolation process X [0, 1]3 rectifiable. In this case, the called the rectifiedflow of coupling (1), denoted as Z = RectFlox X1)), and (Zo, Z1) called the rectified coupling of (Xu, X1), denoted as (Zo, 71) = RectFlow ((X,, X1)).
(tX, + (1 x)Xo:
(Z,: t [0. 1]} in (11) is
Theorem D.5. Assume (Xo, X1) rectifiable and (Zo, Z RectFloo (No, X1)). Then for any
convex function e: RRd R. we have
Elc(Z] 7o)l Elc(Xi Xo))-
24

---

Published as a conference paper at ICLR 2023
Proof.
The proof is based elementary applications of Jensen's 's inequality.
E [
Zo)l =
y (Ze.
l/as dZ, = xx (Ze, t)dt
< E
(v* (2,, 1)) dt
//convexity of Jensen's inequality HIX, and Z, shares the same marginals
(xx
(Xx, 1)) dt
E
c cB[((X, - Xo) X.)) dl
/ldefinition of x
E
B(c(X1 -xo) ( xx]dt
lcconvexity of c, Jensen's inequality
[x Xoo]dl
/E - - Xx) Xx)]
E[c(X1 Xo)l-
If X₁ is straight but with positive non-constant speed, that is, X, = Xi + B.Xo with 8, = 1 or and O 2 then we still have E[c(Z, Zo)] E[c(X1 Xo)] e convex and m-homogeneous in that c(ar) = jal" c(r) for Va E R. E F with some constant m (0,
D.3 THE STRAIGHTENING EFFECT
A coupling (Xo, X1) ) is said to be straight (or fully rectified) if it is fixed point of the RectFlow( mapping. desirable to obtain a straight coupling because rectified flow is straight and hence can be simulated exactly with step using numerical solvers. In this section, the basic properties of straight couplings, showing that a coupling is straight iff its linear interpo- lation paths do not intersect with each other. Then, prove that recursive rectification straightens the coupling and its related flow with a O(1/k) rate, where i is the number rectification steps.
first characterize
Theorem D.6. Assume (Xo, X1) is rectifiable. Let X, = tX1 + (1 - t)X, and Z RectFlow((Xo. X1). Then (Xo, X1) is straight coupling iff the following equivalent statements hold.
1. There exists a strictly corvex function c: Idd + R. such that Ejc(Zi - -oo]l = Efc(X, Xo)].
2. (Xo, X1) is a fixed point of RectFlou()), that is, (Xo, (Zo: Z1).
3. The rectified flow coincides with the linear interpolation process: X Z.
4. The paths of the linear interpolation X do not intersect:
v(Xo,, X1)) = f E |X Xo E E|x1 Xoo
Xe]l| dt 0,
(14)
where V((g, X1)) indicates that X, Xo = FE[X X,] almost surely when Uniform( [0, 1)). meaning that the lines passing through each X₃ is unique, and hence no linear interpolation paths intersect.
Proof. 3 2 1: Obvious.
1 4: If E[c([, - Zp)] ) = Ele(X, - -xo)]. the two applications of Jensen's 's inequality in the proof of Theorem D.5 are tight. Because is strictly convex, the second Jensen's inequality in the proof implies that X, - Xo = E[X, - X, X2] almost surely w.r.t. X and Uniform (0, 1)), which implies that V(X) 0.
25

---

Published as a conference paper at ICLR 2023
4 + 3: (X) = 0, we have (X1 Xo)dt - XooXi]dtt = So xx (Xe, for E (0, 1/. Hence
X, X I (X1 (X1- Xo)dt =xu + S(" xx (X,, t)dt.
Because Z satisfies the same equation (11), have X Z by the uniqueness of the solution.
O(1/K) convergence rate We now show that we apply rectification recursively, the rectified flows become increasingly straight and the linear interpolation of the couplings becomes increas- ingly non- -ntreseeteecing.
Theorem D.7. Let z the k-th rectified flow of (Xo, that is, z RectFlow((Z) zf) and (Zg, 2 = (Xo, X1). Assume each (Zg. 2/) is rectifiable for k O,...,K K.
Then
K s(z ++ + V((2) 2 < E [|xx - Xool2]
h=0
Hence, X| - Xoll3) < +oc, we have mink x (S(Z") + V((2)) =
Proof. Taking e(z) ||=|| in the proof of Theorem 3.5, obtain that
E || Xo || - E Zi Zoll] = S Z) V((Xo. X1)).
(15)
Applying it to each rectification step yields
E
2 |] - E | -
- ||]]] -
S(Z + V ((2,22))
A telescoping sum on k = O..... K gives the result.
D.4 STRAIGHT VS. OPTIMAL COUPLINGS
A coupling (Xo.X) is called c-optimal if it achieves the minimum of E[c(X; - Xo)] among all couplings that share the same marginals. Understanding and computing the optimal couplings have been the main focus of optimal transport (e.g., Villani, 2009; Ambrosio et al., 2021; Figalli different desirable property. In the & Glaudo, 2021; Peyré al., 2019). Straight couplings following, we show that straightness is a necessary but not sufficient condition of being c-optimal for a strictly convex function c, except in the one dimensional case when the two concepts coincides. Hence, it is "easier" to find a straight coupling than a c-optimal couplings.
Theorem D.S. If rectifiable coupling (Xo, X1) is c-optimal for some strictly convex cost function then (Xo, X1) is straight coupling.
Proof. Let (Zn, 71) = RRectFlox((X))) X1)). If (Xo, X1) is c-optimal, we must have Efo( Z 7o) Blc(X: Xo)]. This implies Statement 1 in Theorem D.6 and hence that (Xo, X1) is straight.
1D Case For any To, Ti on R, there exists an unique coupling (X₂. Xi) that is simultaneously optimal for all non-negative convex cost functions e. This coupling is uniquely characterized by monotonie property: for every (In, 01) and (a/, x1) in the support of (X, Xi), if wo < 2/- then Ii I x1. Furthermore, if xo is absolutely continuously w.r.t. the Lebesgue measure, then Xx.x1) must be deterministic that there exists a mapping T:: R + R such that Xi = T(X5). See Santambrogio (2015).
In the following, we show that straight couplings on R coincides with the deterministic monotonic coupling (Xz.X;) and hence is unique and simultancously optimal for all convex when no is absolutely continuous. The idea is that, in R, a coupling is monotonic iff its linear interpolation paths do not intersect, characteristic feature of straight couplings.
26

---

Published as a conference paper at ICLR 2023
Lemma D.9. A coupling on I is straight iff it is deterministic and monotonic.
Theorem D.10, For any No, on I., there exists either no straight coupling, or a unique straight coupling. Further, if exists, the unique straight coupling deterministic and monotonic, and jointly optimal w.E.. all convex cost functions e: R - (00) for which the minimum value of (c(X1- (6X1-Xo) Xo)] exists and is finite.
Proof of Lemma D.9. If (1) on R is straight, then it coincides with its rectified coupling (Zo. Z1) = RectFlou((Xi)
). But because (Zu, Z1) is induced from the rectified flow dZ. v (Ze, ()d,, it is obviously deterministic. It is also monotonic due the non-crossing property of flows. Specifically, if (Zo, Z1) is not monotonic, there exists (zo, 21) and (1):=1) in the support of (Zo. Z1) such that 20 < 2 and 21 2/. If this happens, there must exists to (0, 1), such that 21 = 2/6 But by the uniqueness of the solution, we have 2 = 2ς for t 2 to. which is conflicting with 21 > zí.
Assume (Xo, X1) is deterministic and monotonic. Due to the monotonicity, there exists no Ty and To in the support of To. such that 2e * zy and 2t, = 4/1y for some to < 1. This suggests that X1 - Xo- = E[X; - Xo Xx] = xx (X,,t) for E (0, and hence dX, = (X1 - -o)dt = v (Xt)dt, which is the ODE of the rectified flow. In addition, X, is obviously the unique solution
of this ODE. Hence (Xo. X1) is rectifiable and straight following Statement 3 of Theorem D.6. O
Proof of Theorem D.10. This the result of Lemma D.9 combined with the fact that the mono- tonic coupling is unique and jointly optimal for all convex for which the optimal coupling exists, following Lemma 2.8 and Theorem 2.9 of Santambrogio (2015).
Multi-dimensional cases On the other hand, on Fd with d 2 2, the different cost functions do not share a common optimal coupling in general, and a straight coupling is not guaranteed to optimize specific e; this is expected because the RectFlow()) procedure does not depend on particular choice of c. Hence, one must modify the RectFlow() procedure to tailor it to a specific c of interest.
In a recent work (Khrulkov & Oseledets, 2022), it was conjectured that the couplings (Zo, Zi) induced from VP ODE (equivalently DDIM) yields an optimal coupling w.r.t. the quadratic loss, which was proved to be false in Lavenant & Santambrogio (2022); Tanana (2021). Here we show that even straight couplings are not guaranteed be optimal, not to mention that VP ODE does not follow straight paths by design.
We explore this in a separate work (Liu, 2022b) that is devoted to modifying rectified flow to find c-optimal couplings, result that can be easily stated is that the optimal coupling w.r.t. the quadratic cost c(-) 11-11² can be achieved as the fixed point of (ectFllo(() if v is restricted to be gradient field of form v(x t) = fz(z, t) when solving (1). Restricting v to be a gradient field removes the rotational component of the velocity field x. that causes sub-optimal transport cost.
D.5 ADDITIONAL TOy EXAMPLES
To accurately illustrate the theoretical properties, we use the non-parametric estimator (xx((z))) (2) in (7) in the toy examples in Figure 2, 3. 11, 12. In practice, we approximate the expectation in (7) an nearest neighbor estimator: given a sample (ar() 210);
(0); drawn from X1), we estimate X by
(8)
X. (2,1)
Σ
wh (n :) / Σ
wh (wt (6) 2),,
ot = 1x + 1)=r
xEknn(z.m)
sEknn(z.m)
where knn(z, 2, m) denotes the top m nearest neighbors of z in {x, not sensitive to the choice of m and the bandwidth h (see Figure 14). We use h = 1 and m 100 by default. The flows are simulated using Euler method with a constant step size of 1/N for N steps. We use N = 100 steps unless otherwise specified.
(0) h. We find that the results are
Alternatively, x can be parameterized as a neural network and trained with stochastic gradient de- scent or Adam. Figure 14 shows an example of when x is parameterized as an 2-hidden-layer fully connected neural network with 64 neurons in both hidden layers. We see that the neural networks fit.
27

---

Published as a conference paper at ICLR 2023
less perfectly with the linear interpolation trajectories (which should be piece- wise linear in this toy example). As shown in Figure 14, we find that enhancing the smoothness of the neural networks (by increasing the L2 regularization coefficient during training) can help straighten the flow, in addition to the rectification effect.
I-Rectified Flow
2-Rectified Flow
3-Rectified Flow
I-Rectified Flow
2-Rectified Flow
3-Rectified Flow
1.2 Penalyyd)
1.2 Penalty = 0.01
Figure 14: Rectified flows fitted with neural networks trained with different L2 penalty (left), and kernel esti- mator with different bandwidth h (right). Toi red dots; Ni: purple dots.
In Figure 3 of Section 2.2, the straightness is calculated as the empirical estimation of (3) based on the simulated trajectories. The relative transport cost is calculated based on {=0)
(1) [i) in
21 Jiz= drawn
from (Zo 71) by simulating the flow, as 1 E
(0) =1= =00 where 2 (t") the
optimal L2 assignment of z obtained by solving the discrete L2 optimal transport problem between { and }- We should note that this metric is only useful in low dimensions, as it tends to be identically zero in high dimensional cases even x is set to be a random neural network. This misleading phenomenon is what causes Khrulkov & Oseledets (2022) to make the false hypothesis that DDIM yields L2 optimal transport.
E ADDITIONAL EXPERIMENT DETAILS
Experiment Configuration on CIFARIO We conduct unconditional image generation with the
CIFAR-10 10 dataset (Krizhevsky et al., 2009). The resolution of the images are set to 32 x 32. For rectified flow, adopt the same network structure as DDPM+ in (Song al., 2020b). The training of the network is smoothed by exponential moving average as in (Song et al., 2020b), with ratio of 0. 0.999999. We adopt Adam (Kingma Ba, 2014) optimizer with learning rate of 2e 4 and dropout rate of 0)115.
For reflow, we first generate million pairs of (zp, 21) get new dataset D, then fine-tune the i-rectified flow model for 300, 000 steps to get the (2 + 1) )-rectifieed flow model. We further distill these rectified flow models for few-step generation. To get k-step image generator from the i- rectified flow, we randomly sample E (0, 1/k, ... (k 1)/k} during fine-tuning, instead of randomly sampling E [0, 1]. Specifically,, for & = 1, we replace the L2 loss function with the LPIPS similarity (Zhang et al., 2018) since it empirically brings better performance.
Experiment Configuration on High-resolution Image Generation In nigh-resolution image generation, we adopt the NCSN+ network, following (Song et al., 2020b). The other configura- tions are kept the same as CIFARIO. We provide additional quantitative results in Table Because the dataset only contains < 6000 images, FID is computed with 5000 images.
Expreiment Configuration on Image-to-Image Translation In this experiment, we also adopt the same U-Net structure of DDPM++ Song et al. (2020b) for representing the drift X We follow the procedure in Algorithm I. For the purpose of generative modeling, we set Ro to be one domain dataset and the other domain dataset. For optimization, 2017) optimizer with /3 (0.9, 0.999), weight decay 0.1 and dropout rate 0.1. We train the model with batch size of 4 for 1, 000 epochs. We further apply exponential moving average (EMA) optimizer with coefficient 0.9999. We perform grid- on the learning rate from (5 x 10 4,2 x 10 4, 4,5x 10-5 2 x 10-5 5 10-6) and pick the model with the lowest training loss.
use AdamW (Loshchilov & Hutter,
28

---

Published as a conference paper at ICLR 2023
Method
NFE())
FID ())
ODE
One-Step Generation (Euler solver, N=
1-Rectified Flow (+Distill) 2-Rectified Flow (+Distill)
227.82 (25.38)
167.79 28.60)
ODE
Full Simulation (Runge Kutta (RK45), Adaptive N)
I-Rectified Flow
201
13.71
2- Rectified Flow
166
20.67
Table 2: FID on AFHQ-CAT dataset.
Method
ERM
IRM
ARM 64.8 +
Mixup
MLDG
CORAL
Ours 69.2 - 0.5
OfficeHome
66.5 =
2.2
68.1 0.3 39.2 0.1
66.8 + 0.6
68.7 + 0.3
DomainNet
10.9 + 0.1
33.9
35.5 + 0.2
11.2 +
41.5 + 0.2
41.4 +
Table 3: The accuracy of the transferred testing data using different methods, on the OfficeHome and Domain- Net dataset. Higher accuracy means the better performance.
We use the AFHQ (Choi et al., 2020), MetFace (Karras et al., 2020) and CelebA-HQ (Karras et al., 2018) dataset. Animal Faces HQ (AFHQ) is an animal-face dataset consisting of 15,000 high-quality images at 512 x 512 resolution. The dataset includes three domains of cat, dog, and wild animals, each providing 5000 images. MetFace consists of 1,336 high-quality PNG human-face images at 1024 x 1024 resolution, extracted from works of art. CelebA-HQ is human-face dataset which consists of 30,000 images at 1024 1024 resolution. We randomly select 80% as the training data and regard the rest the test data, and resize the image to 512 x 512.
E.1 DOMAIN ADAPTATION
A key challenge of applying machine learning to real-world problems is the domain shift between the training and test datasets: the performance of machine learning models may degrade significantly when tested on a novel domain different from the training set. Rectified flow can be applied to transfer the novel domain (mg) the training domain (π1) to mitigate the impact of domain shift.
Experiment settings We test the rectified flow for domain adaptation DomainNet (Peng et al., 2019) is a dataset of common objects six different domain taken from DomainBed (Gulrajani & Lopez-Pazz 2020). All domains from DomainNet include 345 categories (classes) of objects such as Bracelet, plane, bird and cello. Office-Home (Venkateswara et al., 2017) is a benchmark dataset for domain adaptation which contains 4 domains where each domain consists of 65 categories. To apply our method, first we map both the training and testing data to the latent representation from final hidden layer of the pre-trained model, and construct the rectified flow on the latent representation. We use the same DDPM++ model architecture for training. For inference, we set the number of steps of our flow model 100 using uniform discretization. The methods are evaluated by the prediction accuracy of the transferred testing data on the classification model trained on the training data.
number of datasets.
Experiment Configuration For training the model, we apply AdamW (Loshchilov & Hutter, 2017) optimizer with batch size 16, number of iterations 50k, learning rate 10-4, weight decay 0.1 and OneCycle (Smith & Topin, 2019) learning rate schedule.
Results As demonstrated in Table 3, the 1-rectified flow shows state-of-the-art performance on both DomainNet and OfficeHome. It better on par with the previous best approach (Deep CORAL (Sun & Saenko, 2016)). while sustainably improve over all other methods.
29

---

Published as a conference paper at ICLR 2023
Image
Crossover
a 0.9
0.8
= 0.7
0.6
Q 0.5
a/ a.0
Image 2
Reconstruction
Q = 0.9
a = 0.8
a == 0.7
a = 0.6
at = 0.5
* - 02i + vI dt with Ç ((0,
Figure 15: An example of image editing using I-rectified flow. Here, we stitch the images of white cat and black cat into unnatural image (denoted 21). We simulate the ODE reversely from get the latent code 2o. Because 21 not natural image, zo should have low likelihood under ло = =O, 1).. Hence, we move 20 towards high probability region of no to get z/ and solve the ODE forwardly to get more realistically looking image 21. The modification can he done deterministically by improving the rg-likelihood via 2 a00
with 1). or stochastically by Langevin dynamics, which yields formula with { NO, 7).
a + vI a²g
Figure 16: To visualize the latent space, we randomly sample and 2 from (0, 7), and show the generated images of /azo vI az for E 1].
30

---

Published as a conference paper at ICLR 2023
I-Rectified Flow
RX44
Mask
Pattem
2-Rectified I-Reetified
Flow
Distilled
( 11
Flow
2-Rectified Flow
(b)
RXC4S
An
1.0g Zp
1255
1.5c
2-Reetified 1-Itectifiel
Flow
Distileed
Flow
Figure 17: (a) We compare the latent space between Rectified Flow (0) and (1) using different sampling strate- gies same random seeds. observe that (i) both I-Rectified Flow 2-Rectified Flow can provide a smooth latent interpolation, and their latent spaces look similar; (ii) when using one-step sampling (NV = L), 2-Rectified Flow can still provide visually recognizable interpolation, while I-Rectified Flow cannot; (iii) Dis- tilled one-step models can also continuously interpolate between the images, and their latent spaces have little difference with the original flow. We composite the latent codes of two images by replacing the boundary black cal with white cat, then visualize variation along the trajectory. The black cat turns into a grey cat at first, then a cat with mixing colors, and finally a white cat. (c) We randomly sample & ~ (00 I), then generate images with af to examine the influence of a the generated images. We find < 1 results in overly smooth images, while 1 leads noisy images.
I-Rectified Flow
2-Rectified Flow
Z1
Zt
0
0.001 0.002
0.1
0.3
0.5
0.8
0.001 0.002 0.1
0.3
0.5
0.8
41
SubVP-ODE
24
Zg
0.001 0.002
0.3 0.5
Figure 18: Sample trajectories 2t different flows the CIFARIO dataset, and the extrapolation of 2e + (1 f)v(2,, f) from different 2, The same random seed adopted three methods. The of 2-rectified flow is almost independent with f, indicating that its trajectory is almost straight.
31

---

Published as a conference paper at ICLR 2023
21 + A.:-AR-K RKK45
2: -1 RK:4
2u-7 Aa:0
P1 fo: Dotiled
Pu Ro-RK4S RK45
Noillel Cheillll
Ro mb 2y: R0045
By: 10-45
An m> 1p:00-1
Ag mi 8y:00
Rn mi Ry:
Bo m By: Datilled
|-Rectifed
Original Enagge 21
2-Roctiffed
We perform latent space embedding image reconstruction here. Given an image 21, we use an re- verse ODE solver to get a latent code 20, then use forward ODE solver to get reconstruction & of the image. The columns in the figure are reverse ODE solver (forward ODE solver). (i) Thanks to the 'ttraightening' effect, 2-rectified flow can get meaningful latent code with only images using one forward step. (ii) With the help distilled models, one-step embedding and reconstruction significantly improved.
Figure 19:
step.
also generate recognizable
I-Rectified Flow
Rectified Flom
2-Rectified
0
(a) I-rectified flow between different domains
(b) and 2-rectified flow for MetFace Cat.
Figure 20: Samples of trajectories 2, of and 2-rectified flow for transferring between different domains.
32

---

Published as a conference paper at ICLR 2023
Figure 21: More results for image- to image translation between different domains. The images in each row are
time-unifomlly sampled from the trajectory of 1-rectified flow solved N 100 Euler steps with constant step size.
33


Reviews:
[{"review_id": "tZy1afKeP8X", "reviewer": "Reviewer_e8Br", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn continuous-time transport maps defined using ODEs to map between two distributions. The algorithm aims to regularize for straightness of the paths, which can help make simulation faster. The idea of Rectified Flow is to sample pairs (x0, x1) and fit a vector field to the linear interpolation between x0 and x1. This optimization process is then repeated by creating pairs of samples from the model.\n\nThe authors make a series of theoretical statements regarding this training process, especially that if this process is repeated K->inf times, then the paths become straight.\n\nThe experimental results show that the model is competitive in terms of FID with previous diffusion SDE models while being better than deterministic ODE counterparts.", "summary_of_the_review": "This paper makes a number of interesting contributions that I think are useful for the ICLR community, so I vote for acceptance. I do think the writing can be made clearer, especially the connection between the theoretical details and the actual algorithm. How this explicitly relates to OT is also not completely clear, and is only hinted at in scattered places in the paper. Empirical results feel mainly qualitative, and I think (i) reporting NLL and (ii) doing more quantitative comparisons with existing models like GANs will make the results better.", "strength_and_weaknesses": "Strengths:\n  - The objective is simple and achieves reasonable results.\n\nWeaknesses:\n  - The theoretical details of the paper are difficult to follow. It would be good to have a clearer connection/derivation between the theoretical motivation (straight paths) and the objective itself.\n  - Only image quality metrics are reported. Existing ODE/SDE models tend to report NLL in addition to FID, and it would be good to have NLL comparisons. This reviewer's experience with FID has been that it is a rather fickle metric, and FID on CIFAR-10 only compares to training data with no test for generalization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZxkFTIsfPD", "reviewer": "Reviewer_ot8j", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, the authors introduce a method to train a continuous normalizing flow connecting two empirical distributions. They call their flow models rectified flows. These flows are trained by minimizing a simple least square objective given pairs of samples. Thereby, the flow is forced to approximately linearly interpolate between the samples. By iteratively retraining the model (reflow) the rectified flow learns to connect corresponding areas in the distribution.\nThe method is applied to several image generation benchmarks. The quality of the generated images is competitive with other procedures while being very efficient in sampling.", "summary_of_the_review": "Overall, I tend towards accepting this paper, but ask the authors to restructure their paper and put a larger emphasis on their method being useful for domain adaptation.\n\nEdit after rebuttal:\n\nI appreciate the author's effort to improve the readability and theoretical soundness of their work. Therefore, I am in favor of accepting the article and raise my score.", "strength_and_weaknesses": "### Strengths\n\nThe idea of transporting samples of one empirical distribution to another with a continuous normalizing flow is novel and creative. The learning algorithm is intuitive and thoroughly analyzed. Although being counterintuitive at first glance, the method is useful for many applications, the most obvious being domain adaptation like image-to-image translation, but also image generation from noise with improved sampling efficiency.\n\n### Weaknesses\n\nIn my point of view, the most interesting application of this method would be domain adaptation, e.g. changing the weather or light conditions of the images captured by a (self-driving) car. This application is not explored in the paper. Instead, the authors focus on translating humans to cats, which might be amusing but not very useful.\nFurthermore, the structure of the paper is poor. I elaborate on this below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NxBnWpiyxnG", "reviewer": "Reviewer_T1Pp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces rectified flow, which is an ODE model that connects two distributions and samples can be drawn by solving the ODE in two directions. In particular, the ODE aims to model the linear interpolation between samples from two distributions. Techniques like iteratively learning more ODEs and distillation are also introduced. Compared to SDE/ODE of diffusion based models, rectified flow has more linear paths, making numerically solving the ODE and sampling from it easier. Results show some improvements over diffusion based models, and its capability of image-to-image translation.", "summary_of_the_review": "I think this paper would be a good contribution to the community. It has some issues in explaining the idea in an intuitive way, i.e., how the formulation is came up with, and I will increase the score if the presentation become better. ", "strength_and_weaknesses": "Strength: \n\n1. The idea is simple but effective. The proposed ODE formulation is very simple, by learning the direction of linear interpolation between samples from two distribution along the interpolation path. However, experimental results show that this method is very effective. \n\n2. It provides a new perspective of diffusion based models which are currently popular. The formulation itself is highly related to diffusion models. Actually, in the VP-SDE's formulation, the forward SDE looks like a spherical interpolation of data and noise. Here the model is doing linear interpolation, and the formulation only requires an ODE. Therefore, it shed new lights on studying diffusion models.\n\n3. It provides a unified framework for generation and domain translation, which is an important contribution.\n\nWeakness:\n1. Although the ODE formulation and training objective is simple, I do not have a good intuitive understanding of choosing this ODE formulation. I found the sentence \"To understand the method intuitively...\" at the end of page 3 not helpful at all. A detailed explanation of why choosing this ODE formulation is necessary in the main text. \n\n2. In addition, there seems not to have clear explanation on why reflow will make the flow more straight. It is not trivial to understand this. \n\n2. Since one major selling point of this method is that it takes less NFEs to sample from, a more comprehensive comparison between existing related approaches on diffusion models is needed. For example, in table 1, I think results of DDIM (not the 1-step distillation) and progressive distillation [1] should be included. These results will better show the trade-off between NFE and FID. In particular, progressive distillation also does very well when NFE is low. Since distillation is also an (optional) part of your method, the comparison is even more important.\n\n[1] Progressive Distillation for Fast Sampling of Diffusion Models, https://arxiv.org/abs/2202.00512", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%