FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner
Authors: Wenliang Zhao
Minglei Shi
Xumin Yu
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: flow-based generation
diffusion models
efficient sampling
image generation
Primary area: diffusion_based_models
TL;DR: We propose a framework called FlowTurbo to accerate flow-based generative models by 29.8%-58.3%, which is efficient in both training (<6 GPU hours) and inference (~40ms / img)

Abstract: Building on the success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc. By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1\%$\sim$58.3\% on class-conditional generation and 29.8\%$\sim$38.5\% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-art. Code is available at https://github.com/shiml20/FlowTurbo.

FlowTurbe: Towards Real-time Flow-Based Image Generation with Velocity Refiner
Wenliang Zhao"
Minglei Shi' Department of Automation Tsinghua University stephenserryle @gnail.comm
Department of Automation Tsinghua University mon@gmail.com com wenliangzhao thu@gmail s
Xumin Yu Department Automation
Jie Zhou
Jiwen Lu Department of Automation Tsinghua University luj inj@@tsinggha.a edu, cn
Department Automation Tsinghua University jzhouttsinghua edu. on
Tsinghua University
yuxumin980gmii..cmm
Abstract
Building on the success diffusion models in visual generation, flow-based models reemerge another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce straighter sampling trajectory. which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed efficient sampling of flow-based generative models has been rarely explored. In this paper. we propose a framework called FlowTurbo accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation that the velocity predictor's outputs the flow-based models will become stable during the sampling, enabling the estimation velocity via lightweight velocity refiner. Additionally, we introduce several techniques including pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it be effectively applied for various tasks such image editing, inpainting, etc. By integrating Flow Turbo into different flow-based models, we obtain acceleration ratio of 53.1 58..%% 99.8%~38.54 on text-to-image generation. Notably, Flow/Trre reaches an FID of 2.12 ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-arr, Code is available at hattps://githul com shtsshh/mmmm/c///m.. Se11mm12//FooTurbo FlowTurbo
dlass-oonditionll generation and
Introduction
In recent years, diffusion models have emerged powerful generative models, drawing considerable interest and demonstrating remarkable performance across various domains 10 38 30 12]. Diffusion models utilize denoising network, Cg. to learn the reverse of diffusion process that gradually adds noise to transform the data distribution into a Gaussian distribution. While the formulation of diffusion models enables stable training and flexible condition injection (30), sampling from these models requires iterative denoising. This process necessitates multiple evaluations of the denoising network, thereby increasing computational costs. To address this, several techniques such fast
"Equal contribution. *Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

diffusion samplers 22 18 42 and efficient distillation 31 37 have been proposed to reduce the sampling steps of diffusion models.
Alongside the research on diffusion models, flow-based models 5 19 17 have garnered increasing attention due to their versatility in modeling data distributions. Flow is defined probability path that connects two distributions and can be efficiently modeled by learning a neural network to estimate the conditional velocity field through a neural network Va via flow matching [17). Encompassing the standard diffusion process as a special case, flow-based generative models support more flexible choices of probability paths. Recent work has favored a simple linear interpolant path (20 24 8] which corresponds to the optimal transport from the Gaussian distribution to the data distribution. This linear connection between data and noise results in a more efficient sampling process for flow- based models. However, unlike diffusion models, which benefit from numerous efficient sampling methods, current samplers for pow-based models primarily rely on traditional numerical methods such as Fuler's method and Heun's 's method [24]. These traditional methods, while functional, fail to fully exploit the unique properties of flow-based generative models, thereby limiting the potential for faster and more efficient sampling.
In this paper, we propose FlowTurbo, a framework designed accelerate the generation process of flow-based generative models. FlowTurbo is motivated by comparing the training objectives of diffusion and flow-based generative models, as well as analyzing how the prediction results eœ and Ve vary over time. Our observation, illustrated in Figure [ indicates that the velocity predictions of a flow-based model remain relatively stable during sampling, in contrast to the more variable predictions of in diffusion models. This stability allows us to regress the offset of the velocity at each sampling step using a lightweight velocity refner, which contains only 5% of the parameters of the original velocity prediction model. During the sampling, process, we can replace the original velocity prediction model with our lightweight refiner at specific steps to reduce computational costs.
As a step towards real-time image generation, we propose two useful techniques called pseudo corrector and sample-aware compilation to further improve the sampling speed. Specifically, the pseudo corrector method modifies the updating rule in Heun's 's method by reusing the velocity prediction of the previous sampling step, which will reduce the number of model evaluations at each step by half while keeping the original convergence order. The sample-aware compilation integrates the model evaluations, the sampling steps as well as the classifier-fres guidance [ together and compile them into a static graph, which can bring extra speedup compared with standard model-level compilation. Since each sample block independent, we can still adjust the number of inference steps and sampling configurations flexibly.
framework is fundamentally different from previous one-step distillation methods for diffussin models 20 40 32), which require generating millions of noise-image pairs offline and conducting distillation over hundreds of GPU days. In contrast, Furbo's''s velocity refiner can be efficiently trained on pure images in less than 6 hours. Moreover, one-step distillation bessllattoo--baed methods are limited to image generation and disable most of the functionalities of the original base model. Conversely, FlowTurro preserves the multi-step sampling paradigm, allowing it to be effectively applied to various tasks such as image editing, inpainting. and more.
Our
We perform extensive experiments to evaluate our method. By applying Turbo to different flow-based models, we obtain an acceleration ratio of 53.1 13.11%-58.3% on class-conditional generation and 29.8 19.8%~38.55% on text-to-image generation. Notably, FlowTurbo attains an FID score of 2. on ImageNet with 100 (ms / img) and FID of score 3.93 with 38 (ms img), thereby enabling real- time image generation and establishes the new state-of-the-art Additionally, we present qualitative comparisons demonstrating how FlowTurbo generates superior images with higher throughput and how it can be seamlessly integrated into various applications such as image editing, inpainting, etc. We believe our FlowTurhh can serve as general framework to accelerate flow-based generative models and will see wider use as these models continue to grow 24
20 g 9.
2 Related Work
Diffusion and flow-based models. Diffusion models 100 38
are a family of generative models that have become the de-facto method for high-quality generation. The diffusion process gradually adds noise to transform the data distribution to a normal distribution, and the goal of diffusion models is to
use a network Eg to learn the reverse of the diffusion process via score-matchine [10] (38). Rombach et

---

0.70
0/0
SIT
5IT
Medium
Normalized Value Curvature
SD3- 3D3-Medium
0.53
FLUX
Open-Sora
642
698
0.35
640
600
(b) SiT
(c) SD3-Medium
0.18
FLUX11ddvv
Open-Sora
0.00
000
0.0
12
Sampling Steps
000
0.00
22
22
(a) Comparison of curvatures of different models.
(d) FLUX.I-de
(c) Open-Sora
Figure 1: Visualization of the curvatures the sampling trajectories different models. We compare the curvatures of the model predictions of standard diffusion model (DiT (28) and several
flow-based models (SiT [24], SD3-Medium (8) FLUX.I-dew [I4), and Open-Sora 43) during the
sampling. We observe that the Va in flow-based models is much more stable than of diffusion models during the sampling, which motivates us to seek a more lightweight estimation model to reduce the sampling costs of flow-based generative models.
al. [30] first scales up diffusion models to acage-scale eext-to-image generation by performing the diffusion on latent space and adopting cross-attention to inject conditions. The pre-trained diffusion
models can also be easily fine-tuned achieve generation with more diverse conditions [4 27 and
have attracted increasing attention in the community. Flow-hased generative models are different from
diffusion models both data modeling and training objectives. Flow-based models (20 17
[2]
consider the probability path from one distribution to another, and learn the velocity field via flow matching [17]. By choosing the linear interpolant as the probability path which corresponds to the optimal transport from the normal distribution to the data distribution, the trajectory from noise to data becomes more straighter which is beneficial to the sampling. Recent work [24 g have demonstrates the effectiveness and scalability of flow-based generation models. However, both diffusion and flow-based models requires multiple evaluations of the prediction model, leading to lower inference speed than traditional architectures like GAN. In this work, focus this issue and aim to accelerate flow-based generative models.
Efficient visual generation. Accelerating the generation of diffusion models has become increas- ingly important topic. Existing methods can be roughly categorized as training-free and training-baat -based methods. Training-free methods aim to design faster samplers that can reduce the approximation eтror when sampling from the diffusion SDE or ODE [36 (2 [8 42), while keeping the weights of diffusion models unchanged. Training Traseing-based methods often aim to reshape the sampling trajectory by distillation from the diffusion model [31 40 to achieve the few-step or even one-step generation.
These training-haid based methods usually requires multiple-round of distillation 3I 20 and expensive training resources (e.g., > 100 GPU days in (20)). Besides, the distilled one-step model no longer supports image editing due to the lack multi-step sampling. Although there are a variety of meth- ods for accelerating diffuuson models, there are few fast sampling methods designed for flow-based generative models. Existing flow-based models adopt traditional numerical methods like Buler's method Heun's method during the inference (24). In this work, we provide a framework called Flow Turbo to accelerate the generation of flowebased models by learning a lightweight velocity refiner (which only requires <6 GPU hours) regress the offset of the velocity. Together with other proposed techniques, FlowTurbo addresses the previously unmet need for an efficient flow-based generation framework, paving the way for real-time generative applications.
3 Method
3.1 Preliminaries: Diffusion and Flow-based Models
Diffusion models. Recently, diffusion models 38 35 30
have emerged as a powerful family of generative models. The diffusion models are trained to learn the inverse of a diffusion process such
3

---

that it can recover the data distribution Po(Xo) from the Gaussian noise. The diffusion process can be represented as:
X 02 OyE, e [0,1], e~ N~,
(I)
where (2t, ot are the chosen noise schedule such that the marginal distribution (x1) (O, I). The optimization of diffusion models can be derived by either minimizing the BLBO of the reverse process [T0] solving the reverse diffusion SDE [38]. which would both lead to the same training
objective of score-matching, i.e,, to learn a noise prediction model E (X,, t) to estimate the scaled score function Vx Pt
LDM (0) (x. (xo)
((1) (xt, () + o Vx log (x
where X(t) is time-dependent coefficient. Sampling from a diffusion model can be achieved by solving the reverse-time SDE or the corresponding diffusion ODEs 38 which can be efficiently achieved by modern fast diffusion samplers (36 22
Flow-based models. Flow-based models can be traced back to Continuous Normalizing Flows [S] (CNF), which is a more generic modeling technique and can capture the probability paths of the diffusion process well [17]. Training CNF becomes more practical since the purpose of the flow matching technique T7]. which learns the conditional velocity field of the flow. Similar to ① we
add some constraints the noise schedule such that Org 1, og and an 0, and then define the flow as:
d (+le) X + 0x + orE,
In this case, the velocity field that generates the flow the can represented as:
d (xole) x,xo + Ôye.
ue (x dt
(4)
The training objective of conditional flow matching is train velocity prediction model va to estimate the conditional velocity field:
| vo(
(FM (e))
de (xole)
(5)
The sampling of a flow-based model can be achieved by solving the probability flow ODE with the learned velocity
dx; dt
(x),,t)
X1 (x
(6)
Since the formulation of the flow a/e can be viewed as the interpolation between Xo and v, is also referred to as interpolant in some literature || 24. Among various types of interpolants, a very simple choice is linear interpolant (24] 8 where a₂ = (1-t) 11-t) t) and o; = t. In this case, the velocity field becomes straight line connecting the initial noise and the data point, which also corresponds to the optimal transport between the two distributions [7] The effectiveness and scalability of the
linear interpolant have also been proven in recent work [7 24 g
3.2 Efficient Estimation of Velocity
We consider the velocity estimation in flow-based generative models with the linear interpolant 24 20 As shown in (5). the training target of the velocity prediction model va is exactly Xu, constant value independent of t. Our main motivation is to efficiently estimate the velocity during the sampling, instead of evaluating the whole velocity prediction model Vo every time.
Analyzing the stability of velocity. We start by analyzing the stability of the output value of va along the sampling trajectory. By comparing the training objectives of diffusion and flow-based models 3). we know that the target of Va is independent of t. A more in-depth discussion is provided Appendix A..] where we show the two training objectives have different time- demendenn weight functions. To verify whether there are similar patterns during the sampling, we compare how the prediction results change across the sampling steps in Figure Specifically, we compare the curvatures of of a diffusion model (DiT 28 ) and the Va of flow-based models (SiT (8) SD3 [8]. etc) during the sampling steps. For each model, we sample from random noises and set the total sampling steps as 20. It can be clearly observed that vo of a flow-basee model is much more stable

---

(a) Learning Lightweight Velocity Refiner
(b) Heun's 's Method Sample Block
Xy ~ Po (X), (0.
Mer, (Nole) = (1 4-13%0 + f-16
Velocity Predictor Ve
X
Velocity Predictor V
Velocity Refiner ri
dr
Vig-1
Velocity Predictor Vi
6 18 19, vo (o
X
(c) F FooTuuro Sampling
(c) Pseudo Corrector Sample Block
N = My Mp Nn
No
Xeyu 1
Velocity
Heun's Method
Pseudo Corrector Sample Block
Velocity Refiner Sample block
Cachu
CLS
Sample Block
SAC
SAC
SAC
Velocity Predictor V.
x 1(01))
SAC: Sample-Aware Compilation
X
Figure 2: Overview of Furwoo (a) Motivated by the stability of the velocity predictor's outputs during the sampling, we propose learn lightweight velocity refiner to regress the offset of the velocity field. (b)() We propose the pseudo corrector which leverages a velocity cache reduce the number of model evaluations while maintaining the same convergence order as Heun's method. (d) During sampling, we employ combination of Heun's 's method, the pseudo corrector, and the velocity refiner, where each sample block is processed with the proposed sample-aware compilation.
than the co of a diffusion model. Therefore, We define the Vo as a "stable value". The stability of vo makes possible to obtain the velocity more efficiently rather than performing the forward pass of the whole velocity prediction network Vo at every sampling step.
Learning lightweight velocity refiner. Since the velocity in flow-based model is "stable value". we propose to Iearn a lightweight refiner that can adjust the velocity with minimal computational costs. The velocity refiner takes as inputs both the current intermediate result and the velocity of the previous step, and returns the offset of velocity:
V4.
[X( V4 1..1. + V
(7)
The velocity refiner rs can be designed to be very lightweight (< 5% parameters of vo). The detailed architecture can be found in Appendix C
To learn the velocity refiner, we need to minimize the difference between the output of rg and the actual offset Ve, Vt,-,- However, it requires multiple-step sampling obtain intermediate result Xt, to make the training objective perfectly align with our target. To reduce the training cost, we simulate the x; with one-step sampling starting from Xt,-,, which is directly obtained by the flow wh.... The detailed procedure to compute the loss is listed as follows:
x,-1 < Vurii (xole), Xo Po(x), E px
(8)
Vt.. vo(xx... X2, + Solver Vi..i At)
(9)
C. < [o),,o (ro(xt;; Vt;-1, + v1,-,)11
(10)
Where At 3 t; 4 and we use simple Euler step for the Solver to obtain Xt,- Once the velocity refiner is learned, we can use it to replace the original Va some specifie sampling steps. We will demonstrate through experiments that adding the velocity refiner can improve the sampling quality without introducing noticeable computational overhead.
Compatibility with classifier-free guidance. Classifier-free guidance [I] is a useful technique to improve the sampling quality in conditional sampling. Let y be the condition, the classifier-free guidance for velocity prediction model [8] can be defined as:
vc (x, f (1 (xvx (1Ø) + (vo(x,
(11)
where Ç the guidance scale and Ø denotes the null condition. To make our velocity refiner support classifier-free guidance, we only need to make sure both the conditional prediction ve(x, tly) and the
5

---

appear during the training. Note that we always feed the velocity prediction model Vo and the velocity refiner F with the same condition.
unconditional prediction (x t
Yr Iysn Ø + la>n
7 1],
(12)
C f E| (x 41y))) (Te (Xe. Ve, 2: 4,) Va, .)le
(13)
where we set the 7/1 0.1 as the probability of using an unconditional velocity.
3.3 Towards Real-Time Image Generation
The sampling costs of a flow-based model can be significantly minimized by integrating our lightweight velocity refiner r in place of the velocity prediction network va at selected sampling steps. this section, we propose two techniques to further improve the sampling speed towards real-time image generation.
Pseudo corrector. Traditional numerical ODE solvers are usually used to sample from a probability flow ODE. For example, SiT 8 adopt Heun method (or improved Euler's method) (15) as the ODE solver. The update rule from 1i-1 to 4 can be written as:
d vo 1ly)
X1,
Xt,-1 + Atd;-1
(14)
Al
d < vo(K,,, fly),
d + d.]
(15)
Each Heun step contains a predictor step (14) and a corrector step (5). thus includes two evaluations
of the velocity predictor va. bringing extra inference costs. Motivated by 442, we propose reuse
the d, in the next sampling step, instead of re-computing it via d; <- (x.,
(see Figure
(b)(c) for illustration). We call this pseudo corrector since it is different from the predictor-correcto solvers in numerical analysis. can be proved (see Appendix ( that the pscudo corrector also enjoys 2-order convergence while only having one model evaluation at each step.
Sample-aware compilation. Compiling the network into a static graph is a widely used technique for model acceleration. However, all the previous work only considers network-level compilation, ie., only compiling the en or Vg. We propose the sample-aware compilation which wraps both the forward pass of Vo rg and the sampling operation together (including the classifier-free guidance) and performs the compilation. For example, the sample blocks illustrated in Figure 2 (b, c) are compiled into static graphs. Since each sample block independent, we can still adjust the number of inference steps and sampling configurations flexibly.
3.4 Discussion
Recently, there have been more and more training-basse methods (20, 40 32 aiming to accelerate diffusion models or flow-based models through one-step distillation. Although these methods can achieve faster inference, they usually require generating paired data using the pre-trained model and suffer from large training costs (e.g., >100 GPU days in 20)). Besides, one-step methods only keep the generation ability of the original model while disabling more diverse applications such as image inpainting and image editing. In contrast, our Flow Turbo aims to accelerate flow-based models through velocity refinement, which still works in multi-step manner and performs sampling on the original trajectory. For example, FlowTurbe can be easily combined with existing diffusion- based image editing methods like SDEdit (25) (see Section 4.4
4 Experiments
We conduct extensive experiments verify the effectiveness of FlowTurbo. Specifically, we ap- ply FlowTurbo to both class-conditional image generation and text-to-image generation tasks and demonstrate that FlowTurbo can significantly reduce the sampling costs of the flow-based generative models. We also provide detailed analysis of each component of Flow Turbo, as well as qualitative comparisons of different tasks.
4.1 Setups
In our experiments, we consider two widely used benchmarks including class-conditional image gen- eration and text-to-image generation. For class-conditional image generation, adopt transformer-

---

Table 1: Main results. We apply our Flrho on SYT-XL 24 and the 2-RF of InstaFlow [20] to perform cass-condiiional image generation and texl-to-image generation, respectively. The image quality is measured by the FID 5 on ImageNet (256x256) and the FID 5KG on MS COCO 2017 (512x512). We use the suffix to represent the number of Heun's method block (H), pscudo corrector block (P), and the velocity refiner block (R). Our results demonstrate that FlowTurbo can significantly accelcrate the inference flow-based models while achieving better sampling quality.
(a) Class-conditional Image Generation
(b) Text-to-image Generation
Sample
FLOPs (G)
Latency (ms rin))
Sample Config
FLOPs (G)
Latency (ms img)
Method
FID
Method
FID
Config
SIT-X1. 8. ImageNet (256
256)
bsstFFlow
(20), MS CoCO 2017 (512 512)
Heun's
Hy
1898
3.68
89.4
Heun's
Hg
3955
32.77
104.5
FlowTurbo
HrP,Ra
957
3.63
41.6 (-53.4%)
FlowTubo
2649
32.48
(-34
Heun's
H11
2610
2.79
117.8
Heun's 's
Hs
4633
30.73
120,3
FlowTubo
H2P3Ra
1431
2.69
55.2 1-53.1%)
FlowTubo
3327
30.19
84.5 (-29.8%)
Heun's
His
3559
2.42
154,8
Heun's 's
Hs
6667
28.61
170.5
HurTo
Ms
2274
2.22
72.5 (-53.2%)
FloTTubo
P&
4030
28.60
104.8 (-38.5%)
Heun's
Hos
5694
2.20
240.6
Heun's
H10
8023
28.06
203.7
FlowTurbo
H.PoRs
3457
2.12
100,3 (-58.3%)
FlowTuro
HyPoRa
5386
27.60
137.0
(37.0(-32.7%)
Table 2: Comparisons with the state-of-the-arts. We compare the sampling quality and speed of different
methods on ImageNet 256 x 256 class-conditional sampling. We demonstrate that Flow Moobo can significantly
improve over the baseline SiT-XL. 24 and achieves the fastest sampling (38 ms img) and the best quality (2.12
FID) with different configurations
Latency (ms / img)
Model
Sample Contig
Params
FID!
Precision
Recall
StyleGAN-XI
33
166M
190
2.30
265.1
0.78
0.53
Mask-GIT
2
8 steps
227M
120
6.18
182.1
0.80
0.51
ADM 7
250 steps DDIM
56
554M
2553
10.94
101.0
0.69
0.63
ADM-G
250 steps DDIM
36
608M
4764
4.59
186.7
0.83
0,53
LDM-4-G
30
250 steps DDIM
36
400M
448
3.60
247.7
0.87
0.48
DIT-XI. 28
250 steps DDPM
10
675M
6914
2.27
278.2
0.83
0.57
SIT-XL
24
25 steps dopri5
15
675M
3225
2.15
258.1
0.81
0.60
SiT-XL
steps Heun's
15
675M
250
2.20
254.9
0.81
0.60
Hlo Turbo (ours)
PtRRs
704M
38
3.93
223.6
0.79
0.56
FlowTurbs (ours)
HyPPRa
704M
2.22
248.0
0.81
0.60
FlowTurbo (ours)
HaPaRa
704M
2.12
255.6
0.81
0.60
style flowe model SiT-XL [4] pre-trained on ImageNet 256x256. For text-to-image generation, we utilize InstaFlow 220] as the flow-based model, whose backbone a U-Net similar Stable- Diffusion [30]. Note that we use the 2-RF model from [19] instead of the distilled version since achieve acceleration within the multi-step sampling framework. The velocity refiner only contains 4.3% and 5% parameters of the corresponding predictor, and the detailed architecture can be found in Appendix C During training, we randomly sample At E (0, 0.12] and compute the training objectives in (13). In both tasks, we use a single NVIDIA A800 GPU to train the velocity refiner and find it converges within hours. We use a batch size of on a single A800 GPU to measure the latency of each method. Please refer to Appendix C for more details.
our FlowTurbo is designed
4.2 Main Results
Class-conditional image generation. We adopt the SiT-XL (24) trained on ImageNet [6] of resolution of 256 256. Following common practice 24 30), we adopt a classifier-free guidance scale (CFG) of 1.5. According to 24]. widely used sampling method of the flow-based model is Heun's 's method [5]. In Table
Ta we demonstrate how our FlowTurbo can achieve faster inference than Heun's 's method in various computational budgets. Specifically, we conduct experiments with different sampling configurations (the second column of Table la}. where we use the suffix to represent the number of Heun's 's method block (H), pseudo corrector block ( (P), and the velocity refiner block (/t). Note that each Heun's 's block contains two evaluations of the velocity predictor while each pseudo corrector block only contains one. We also provide the total FLOPs during the sampling and the
7

---

inference speed of each sample configuration. In each group of comparison, we choose the sampling strategy of FlowTurbo make the sampling quality (measured by the FID 50K.) similar to the baseline. Our results demonstrate that FlowTurbo can accelerate the inference by 37.2% ~ 43.1%. while still achieving hetter sampling quality. Notably, FlowTurbo obtains 3.63 FID with a sampling speed of 41.6 ms/img, achieving real-time image generation.
Text-to-image generation. We adopt the 2-RF model in [20] as our base model for text-to-image
generation. Note that we do not adopt the distilled version in [20] since focus on accelerating
flow-based models within the multi-step sampling paradigm. Following [20] 26). we compute
the FII 5K4 between the generated 512 512 samples and the images MS COCO 2017 [16]
validation set. The results are summarized in Table Tbj where we compare the sampling speed/quality with the baseline Heun's 's method. Note that the notation of the sampling configuration is the same as Table Ta The results clearly demonstrate that Our Toowo can also achieve significant acceleration 229.8%~3..554 on text-lo-image generation.
4.3 Comparisons to State-of-the-Arts
In Table we compare our FlowTurbo with state-of of the- art methods on ImageNet 256 x 256 class-conditional generation. We use SiT-XL [4 as our base model and apply FlowTurbo with different sampling configurations
SIT
FlowTurbo
Mask-GIT
FID 50K
6-
on it. We show that FlowTurho with H1 Ps R3 achieves
the sampling speed of 38 (ms / img) with 3.93 FID (still better than most methods like Mask-GIT (2), ADM (7)..
5
ADM-G
4
LDM-4-G
On the other hand, FlowTurbo with Hg Mg archives the
3-
lowest FID 2.12, outperforming all the other methods. Besides, also provide comparison of the sampling speed/quality trade-offs of SiT (by changing the number of sampling steps of Heun's 's method) and Flowourro (by changing the sampling configurations) in 3 where the results of some other state-of-the-arts methods also included. The comparison shows our FlowTurbo exhibits favorable sampling quality/speed trade-offs.
StyleGAN-XL
DIT-XL
2-
1
40
100
1600
10000
Latency (ms / img)
Figure 3: FlowTurbo exhibits favorable trade-offs compared with SOTA methods.
4.4 Analysis
Ablation of components of FlowTurbo. We evaluate the effectiveness of each component of FlowTurbo in Table 3a| Specifically, we start from the baseline, 7-step Heun's method and gradually add components of Flow Turbo. In the sample config A, we show that adding velocity refiner can significantly improve the FID. (4.42 + 2.80), while introducing minimal computational costs (only +0.7% in the latency). From B to E, we adjust the ratios of Heun's 's method block, the pseudo corrector block, and the velocity refiner block to achieve different trade-offs between sampling speed and quality. In the last two rows, we show that our sample- aware compilation is better than standard model-level compilation, further increasing the sampling speed.
Choice of AL. We find the choice of AL during training is crucial and affects the sampling results lot in our experiments, as shown in Table 3b We find At e (0.0, 0.1] works well for more sampling steps like H12 while At [0.06. 0.12 is better for fewer sampling steps like HşRo and Hg Besides, we find At E (0.0, 0.12] yields relatively good results in all the situations.
Effects of velocity refiner. We evaluate the effects of the different number of velocity refiners in Table and find that appropriately increasing the number of velocity refiners can improve the trade-off between sampling quality and speed. Specifically, find HeRe can achieve better image quality and generation speed than the baseline Hs.
Effects of pseudo corrector. In Table 3d. we fix the total number of both Heun's sample block and pseudo corrector block and adjust the ratio of the pseudo corrector. Our results demonstrate that increasing the number of pseudo corrector blocks can significantly improve the sampling speed while introducing neglectable performance drop (e.g., Flow with Hi PoRt performs better than Hss).
8

---

Table 3: Ablation studies. We evaluate effectiveness each component in Flow as well as selection of some syper-parameters. (a) We gradually add the components of FlowTurbe to the baseline and show that FlowTurbo can achieve over 50% acceleration with better sampling quality, (b) we experiment with different ranges of At and find At E (0.0, 0.12) yiclds relatively good results in all the situations. (c)(d) show how the sampling quality/speed changes with the number of velocity refiner and pseudo corrector blocks.
(a) Ablation of components of FlowTurbo.
(c) Effects of the velocity refiner.
Sample Config
Latency (ms / img)
Sample Config
Latency (ms img)
Ni
Np
H
FID
FID
baselines
0
0
4.42
80.0
Hs
3.68
68.0
0
0
3.68
89.4 (+1188%)
H,R,
2.80
61.6 (-9.4%)
A BBU PCCE
2.80
80.5 (+0.7%)
HgRa
3,55
55.2
2.69
71.6 (-10.4%)
HaRs
7.62
49.9 (-26.5%)
3
3.25
57.4 (-28.2%)
4
3.63
52.7 (-34.1%)
(d) Effects of the pscudo corrector.
E
5
3.93
48.0 (-40.0%)
E+ Model-Level Comp.
3.93
41.0 (-48.7%)
Sample Config
Latency
FID
E Sample- Aware Comp.
3.93
38.3 (-52.2%6
(ms img)
Hs
3.68
68.0
(b) Ablation of At.
2.93
62.0 (-8.8%)
Sample Config
Ar \ FID]
Pg
2.60
58.6 (-13.8%)
HsP2Ra
2.66
55.2 (-18.8%)
00.0,0.1]
(0.0,0.12)
(0.0,0.2
[0.06, 0.12]
HaP
2.78
51.8 (-23.8%)
HaRa
3,58
3.55
4.48
3,36
HaP
2.96
48.4 (-28.8%)
HgRs
2,73
2.65
2.93
2.62
H2P. Ra
3.21
45.0 (-33.7%)
HyyRs
2.53
2.54
2.64
2.89
Po 3.59
41.6
(-38.7%)
Table 4: Comparisons of different orders of the blocks. We compare the results of changing the orders of Heun's block (H), pseudo corrector block (4"), and the velocity refiner block (R). Our results show applying the blocks in Hng PNp HNg order yields the best trade-off between generation quality and speed.
(a) Altering the order of H H/P/R RR blocks.
(b) Repeating multiple H/P/ blocks
Latency (ms / img)
Latency (ms / img)
Sample Config
FID +
Sumple Config
FID
H, Ra P. P,H,R P,R:H: H3RPP3
2.24
72.4
[[ P; Rilxa H2 Mr R2
8.15
34.8
2.38
79.2
6.39
34.8
13.66
53.7
[1 Pi Ri]x3
4.34
45.4
30.94
60.4
HgP3Rs
3.34
45.4
Ablation of refiner architectures. When designing the architecture for the velocity refiner, we followed a simple rule make the refiner have a similar architecture as the base ve- locity predictor but with much fewer parameters 5% of the base model). The detailed architecture is described in Section [4]]] and Appendix [ For example, since SiT consists of multiple transformer blocks, we simply use single block as the refiner. For text-to-image generation, we reduce the number of layers and channels of the UNet. In early experiments, we have tried another architecture for class-conditional image generation, where a SiT-S (a smaller version of the base velocity predictor SiT-XL) is adopted as the refiner (as shown in Table 5 We find that using a block of SiT-XL the refiner is slightly better than the SiT-S. These results demonstrate that our framework is robust to the choice of model architectures for the velocity refiner.
Table 5: Ablation of refiner architectures.
Sample Config
Architecture
Params
FID +
Ry H3PPRs
SiT-S
33M
2.53
block of SIT-XL
29M
2.22
HsPoRs HgPoRs
SiT-S
33M
2.24
a block of SIT-XL
29M
2.12
Comparisons of different order of the blocks. According to the observation in Figure I the velocity during the sampling would become stable at the final few steps, where we adopted a lightweight refiner to regress the velocity offset. Besides, our pscudo corrector is designed to efficiently achieve

---

wning Eoat naten Nakgram My
mitteciona
allay
cittirg
(a) Results of Heun's (2.6 s/ img, left) and FlowTurbo (1.8 img, right)
(b) Extensions
Figure 4: Qualitative results. (a) We compared our FlowTurbo with Heun's method Lumina-Ness With better image quality, our method requires much less sampling time (-30.8%). (b) Since FlowTurbo remains the multi-step sampling paradigm, it can be seamlessly applied to more applications such as image inpainting, image editing, and object removal.
2-order convergence, which requires a 2-order intermediate result initialization. This explains why we need several Heun's steps at the beginning. To further investigate how the order of the blocks would affect the sampling speed and quality, perform experiments and summarize the results in Tables 4a and 4bb First, we find changing the order of the sampling blocks will cause worse sampling quality. Second, we show that sequentially using multiple blocks (e.g.. [H PNg RNglxa) will cost the same inference time as HEND PANE RENz but lead to worse visual quality.
Qualitative results and extensions. We provide qualitative results of high-resolution text-to-image generation by applying Flow/Trrb to the newly released flow-based model Lumina-Next- [9]. Since Lumina-Next-T21 adopts heavy language model Gemma-21 2B (39 to extract text features and generates high-resolution images (1024 x 1024), the inference speed of it is slower than SiT (24). In Figure 4al we show that our FlowTurbo can generate images with better quality and higher inference speed compared with the baseline Heun's 's method. Besides, since Flow Turbo remains the multi-step sampling paradigm, can be seamlessly applied more applications like image inpainting. image editing, and object removal (Section 4.4). Please also refer to the Appendix C for the detailed implementation of various tasks.
Limitations and broader impact. Despite the effectiveness of Flow our velocity refiner highly relies on the observation that the velocity is a "stable value" during the sampling. However, we have not found such stable value in diffusion-based models yet, which might limit the application. Besides, the abuse of FlowTurbo may also accelerate the generation of malicious content.
5 Conclusion
In this paper, we introduce Flow Torbo, a novel framework designed to accelerate flow-based genera- tive models. By leveraging the stability of the velocity predictor's outputs, we propose lightweight velocity refiner to adjust the velocity field offsets. This refiner comprises only about 5% of the original velocity predictor's 's parameters and can be efficiently trained in under 6 GPU hours. Additionally, we have proposed a pseudo corrector that reduces the number of model evaluations while maintain- ing the same convergence order as the second-order Heun's method. Furthermore, we propose sample-aware compilation technique to enhance sampling speed. Extensive experiments on various flow-based generative models demonstrate FlowTurbo's 's effectiveness on both ilass-condiiioal image generation and text-to-imane generation. We hope our work will inspire future efforts to accelerate flow-based generative models across various application scenarios.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 62321005, Grant 624B 1026, Grant 62336004, and Grant 62125603,
10

---

References
[1] Michael S Albergo, Nicholas M Boffi, and Eric Mijeee- Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXw:2333300879, 2023.
[2] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 11315-11325 2022.
[3] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xianzhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403. 04692, 2024.
[4] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xic, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis, arXiv preprint arXiv:2310.0044
2023.
[5] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[6] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. CVPR, pages 248-255, IEEE, 2009.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8794, 2021.
[8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403. 032006 2024.
[9] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang. et al. Lumina-t2x: Transforming text. into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405. 05544, 2024.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbcel. Denoising diffusion probabilistic models. Neur|PS, 33:6840-
6851, 2020.
[11] Jonathan Ho and Tim Salimans. Classifier- freee diffusion guidance. NeurIPS, 2021.
[12] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.0344 2022.
[13] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Neur!PS,
34:21696-21 21707, 2021.
[14] Black Forest Labs. black-forest- Laats--foortt-La/aLLLL..... dev 2024. Accessed: 2024-09-26.
Flux: A powerful tool for text generation.
httpa://hugginggfaceccm
[15] John Denholm Lambert et al. Numerical methods for ordinary differential systems, volume 146. Wiley New York, 1991.
[16] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[17] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[18] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. ICLR, 2022.
[19] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0304
2022. [20] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high- quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
[21] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05100 2017.
arXiv preprint
[22] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NewrIPS, 2022.
[23] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:22 0ttt:22/1.01095, 2022.
[24] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Bofti, Eric Vanden- Eijnnen, and Saining Xie. Sit: Exploring flow and diffusion based generative models with scalable interpolant transformers. arXiv preprint 240: 08740, 2024.
11

---

[25] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprini arXiv:2108.0107. 2021.
[26] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 14297-1453 14306, 2023.
(27] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang. Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[28] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[29] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dackhorn, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[31] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR, 2022.
[32] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation.
arXiv preprint arXiv:2311.1708 2023.
[33] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets, In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[34] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk. Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 14, 2021.
[35] Jascha Sohl-Dicksteni Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning PMLR, 2015.
using nonequilibrium thermodynamics. ICML, pages
[36] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
[37] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[38] Yang Song, Jascha Dickkiics,,,,,, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modcling through stochastic differential equations. In ICLR, 2021.
[39] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology- arXiv preprint arXiv:2403.088 03.8959 2024.
[40] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2023.
[41] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages
2023.
[42] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-corrector framework for fast sampling of diffusson models. NeurIPS, 2023.
[43] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient vidco production for all, 2024.
12

---

A
Detailed Background of Diffusion and Flow-based Models
In this section, we will provide a detailed background of diffusion and flow-based models, which is helpful to understand the difference and relationship between them.
A.1 Diffusion Models
The forward pass i.c. diffusioin pass of DPMs can be defined as sequence of variables {x1 rejo.1] starting with zo, such that for any & [0, 1J, X Rº is a D-dimensional random variable with an unknown data distribution (x()).. the distribution of xx conditioned on 2 satisfies
Por(x,|xo) N (x o,I)
(16)
where at, Og E R+ are differentiable functions of & with bounded derevatives. The choice for a and is referred to as the noise schedule of a DPM. Let Pe( denote the marginal distribution of xt, DPMs choose noise schedules to ensure the marginal distribution p1 (x1) = =V I) and the signal-to-noise-rato (SNR) a² /o
? is strictly decreasing w.r.t. i. [13] And we have
x
XxXo + o₁E,
[0,
e~,11011)
(17)
Moreover, Kingma et al. 133 prove that the following stochastic differential equation (SDE) has the same transition distribution que (x, (x) as in (T6) for any E [0, 1]:
dx; = (t) + g(t)dwe, E [0, 1], Xo (o(x))
(18)
where w, E DRD is the standard Wiener process. and
dl logar
(²(1) do? d loga
f(t)
oi
(19)
di
Song et al. (38] have shown that the forward process in (8) has an equivalent reverse process from time 1 under some regularity conditions, starting with the marginal distribution pr (xT)):
dx. [J((), ,² (1)
+ g(()dw,,
pr((T)
(20)
where W, DR is standard Wiener process in the reverse time. To solve the reverse process in 20 the only thing we should do is to estimate the score term Vxlogp: (xr) at each time t. In practice. DPMs train a neural network (x parameterized by 0 to estimate the scaled score function:
- Vx (x.). The parameter 0 is optimized by minimizing the following objective [00 38
(f) (xx.))))))) (xtt)lea(xx, + or V log (x,
(21)
where X(t) is a time-dependent coefficient. As co t) can alse be regarded as predicting the Gaussian noise added to x,, usually called the noise prediction model Since the ground truth of Co (x, t) is -a, V=logpt DPMs replace the score function in 20 by co (x,, f)/o, and we refer it as diffusion--aase
generative model. DPMs define a parameterized reverse process (diffusion SDE) from time 1 to 0 starting with (x(x):
[(()xx+ + g2(1) cx 1) de + g())dw, 31 ~ NN(O
dx,
(22)
Samples can be generated from DPMs by solving the diffusion SDE in (2 with numerical solvers.
When discretizing SDEs, the step size is limited by the randomness of the Wiener process. A large step size (small number of steps) often causes non-convergennce especially in high dimensional spaces. For faster sampling, we can consider the associated probability flow ODE (38] which has the same marginla distribution at each time t as that of the SDE. Specifically, for DPMs, Song et al. (38
proved that the probability flow ODE of (22p is
dx: dt
g2(t)
v(x,x))) S(1xx, +
60 (xt,
N( I)
(23)
20,
Samples can be generated by solving the ODE from 1 to 0. Comparing with SDEs, ODEs can be solved with larger step sizes as they have no randomness. Furthermore, we can take advantage of efficient numerical ODE solvers to accelerate the sampling.
13

---

A.2 Flow-based Models
To introduce flow in detail, first we construct time-dependent vector field, u [0. RD + RDD A vector field us can be used to construct time-dependent diffeomorphic map, called flow, Φ: 0, RD _R ,defined via the ordinary differential equation (ODE);
d (xx)) (x())(xo))
(24)
Φx(x)) X0
(25)
Chen al. [5] suggested modeling the vector field ue with neural network ve. which in turn leads to a deep parametric model of the flow Φt. called Continuous Normalizing Flow (CNF). It is more generic modeling technique and can capture the probability paths of diffusion process well. Training a CNF becomes more practical since the propose of the conditional flow matching (CFM) technique 177, which learns the conditional velocity field of the flow.
For generative models, similar to (17) we can add some constraints to the noise schedule such that ag = 1, 0g 0 and aj 0, 1, and then define the flow as:
(/cc)) x
ox o&E
(26)
The corresponding velocity vector field which is used to construct the flow eh can be represented as:
d
ue (VV(xx)))) thr (xole) å,x0 + ō:E
(27)
di
Consider the time-dependent probability density function (PDF) pa (x) of = (e(zo()) ax TGE. Lipman al. [17] proved that the marginal vector field ue that generates the probability path pt satisfies a Partial Differential Equation (PDE) called continuity equation (also transport equation)
d px(x) + Vx (x)()))((x)) =
(28)
dt
Using conditional flow matching technique v(x,, 1) vo(x,,t) by minimizing the following objective
23 can be estimated parametrically as
LFM(()) = Er, (o).)o(m))
vo(xi,t) - dt the (xol()
(29)
1
pr [v((x;;)) d:x0 ô.e
(30)
We refer to (23) as a flow-based generative model. Since we have x2 = why (2oole), the sampling of a flow-based model be achieved by solving the probability flow ODE with learned velocity
dx: dt
vo(se, I1 Pi(2))
(31)
A.3
Relationship Between Diffusion and Flow-based Models
There exists straightforward connection between v(xi,t) and the score term a, Vx px, according to (24].
de x1
àg
v(x;, t) ā;
(-o, Vx logp:(()))
(32)
az
14

---

Algorithm Heun's Method Sampler
Require: timesteps [t:) de oe, xo NO I), velocity prediction model va(x, tly) for i = 0 to N do
At; f,41 t;
d f vo(x,,
Xt.+1 < X; + A/;d;
d:+1 <- vo vo(xt,11 li+1ly yy)
Af d d 1]
Xt1+1 x; 2
end for return: XN
Algorithm 2 Pseudo Corrector Sampler
Require: timesteps (4: Ote ot, NO, I), velocity prediction model vo(x, (ly)
Al < 1 to for i 0 to N do
At, f fi+1 tú
if i = 0 then
d, < vo(x,, tely)
end if
K/11 < x; + At,d;
de41 Vol vol (8 li+ 1l)
Xt,41 x; + Ati
(d: d,-1] 2
end for
return: XN
We can define Se à, d.a., and have cx(x,, t) to estimate -0, V Vxlogpr then derive the relationship between LDM( and (O) We can plug (32) into the loss LF (0) in Equation (30)
(FM(0) = F. (e))
Vo(ox, t) - 6xX0 0tc|12
(33)
de X1 + ((ea(xx,)) t) à.xo - ō,E
(34)
2
àn + GrE0(x1;1) 1) 6xx00 ô,6
(35)
1
(e))
GG, t) - Gell2
(36)
(e)) xx x= t00
? ||o((x;;)) () E
(37)
[o0o xo)
+
(38)
Recall that
(DM = (x(0)) (xo) (A(e) [|Ea 1) + oz Vx lo
(39)
we can see that the difference of (D) and (PM(() during training is caused by the weighted function, which leaving to different trajectories and properties.
B Proof of Convergence of Pseudo Corrector
In this section, we will prove that the proposed pseudo corrector has the same local truncation error and global convergence order as Heun's 's method. The detailed sampling procedure of Heun's 's method
15

---

and pseudo corrector are provided in Algorithm 1 and Algorithm 2 In this section, we use X represent the intermediate sampling result at the t, timestep. and use x x(t;) denote the corresponding ground-truth value on the trajectory. In all the proofs in this section, we omit the condition y for simplicity.
B.1 Assumptions
Assumption B.1.
The velocity predictor vo(x,1) () is Lipschitz continous of constant L w.r.t x.
Assumption B.2. The velocity predictor v t) has at least 2 derivatives and the derivatives are continuous.
d de vo t) and de de vo t)
Assumption B.3. maxosigN-1 = O(1NN), where N the total number of sampling steps.
All the above are common in the analysis of the convergence order of fast samplers 22 of diffusion models.
B.2 Local Convergence
We start by studying the local convergence and Heun's method. Considering the updating from tę to f4+1 and assume all previous results are correct (see the definition of local convergence [15] The Taylor's expansion of x at t; gives:
12 x 2
+ h (1)+
X1, + hxxx
x 33) (L)) + 0(1*).
(40)
6
On the other hand, let Xt.41 be the prediction assuming x; is correct, the updating rule of Heun's method shows:
h
8111 = =x +
d.
+ d.-1]
(41)
2
h; x (t.) + (t.) + h (t))
h (t)+ +0+))((h)))
=xt,
(42)
2
2
xx hxx (t:) + h
h (t.) (t))0(hh)
=
2 (t+)
(43)
Therefore, the local truncation error can be computed by:
h x (t))
T = | Xt.t =
(4)+0(1)) 0(h*)] s C,
(44)
12
which indicates that Heun's method has 2 order of accuracy.
also noted that the local truncation error of the predictor step (which the same as Buler's method) can be similarly derived by:
h || 11 x +(h))11 Czh²²
T = ||
-x,
=
(45)
2
For pseudo corrector, the analysis of local convergence is the same since we need to assume all previous results (including the xt, and dj), which means the local truncation error of pseudo corrector the same as the Heun's 's method.
B.3 Global Convergence
Global convergence for Heun's method. When analyzing global convergence, we need to take into account both the local truncation error and the effects of the error of previous results. According the Lipschitz condition, we have:
[Xx - Xenz S 5(1 (1+bL)|x,, - x4,1 + Czh
(46)
16

---

and
| xtz- ll S (1 + hb )| xx, + hh
|xx -K44t2l + C.h.
(47)
Combining the above two inequalities together, we have
h² lxxx; - Xx. + Cahi
x Xx,1 1 (1 + hL+ +
(48)
2
where C3 hLC2 + C1- Note that |,=% - stoll = (their is no error at the beginning of the 2
sampling), it can be easily derived that
Czlia L
1-12 hL+
|x X4N
|l
2 - - - =
(49)
++ A7A
2
Therefore, we have proven that Heun's method have 2 order of global convergence.
Global convergence for pseudo corrector. The only difference between pseudo corrector and Heun's method is how d is obtained. Pseudo corrector reuse the d, from the last sampling step rather than re-compute as in Heun's 's method. As a result, d; used pseudo corrector is computed on &t, rather than Xt,, which will lead another error term when analyzing the global convergence. Concretely, the global error of pseudo corrector can be computed by:
||x|.. Ktaill S (1 + x xx,1l + hL
xx: + C2hh²
(50)
hL - -x.111 +
lT
hL
||x| Xt.+ s (1+
- | - 81141 || +
X - -x,, + C, (51)
2
2
2
For the sake of simplicity, let á, = |x7; -x+,11 and A; = ||x| -x,.1. Therefore, the above formulas becomes:
Â A+1 S A; + hLå, + C2h²²
(52)
h1 ト ) (1 (1+ 2 A + Cahi.
hL
hL
1+1 S
(53)
By calculating 52
xhL+
53
we have:
hL
hL
(1 + hL
A + +L
< (1+
A +
) + +ILLA, + C2Lh3
2
2
2
3
WL + Q12L2
A ]
=(1 =
3 hT)
A
2
(54)
2
1+
S(1 (1+ ク
ML) A + MLĀ) + C7113
Note that Ao - hLåA = 0. Let A = A + hLL, we have
3 2 +
A s (1+
(55)
Similar to the derivation of 49 we can derive that
A < C&h "LL""" 1) S Coli,,
(56)
which indicates that Therefore we have AN Cghi., and thus the global convergence of pseudo corrector is 2-order.
A < Coh?, hŁY S Coh?.
(57)
17

---

Table 6: Ablation number of the velocity refiners. We change number of velocity refiners and compare the sampling quality of each configuration. We find there exists a optimal number of velocity refiners to achieve the lowest FID.
Method
Sample Config
Ratio of Refiner
FID
Latency (ms / img)
SiT-XL 88, ImageNet (256 x 256)
Flow Turbo
Hg P. Rs Rs
0.26
2.19
100.7
FlowTurbo
0.23
2.12
100.3
Turbo
Hg RoR
0.19
2.18
99.9
FlowTurbo
Hg Ro
0.15
2.15
99.6
FlowTurbo
HaPoRe
0.29
2.25
87.2
FlowTurbo
Hg Po
0.25
2.20
86.8
Turbo
Ha HaPoRf Ra
0.21
2.21
86.4
FlowTurbo
Hg R3
0.17
2.22
86.0
C Implementation Details
Class-conditional image generation. We use the SiT-XL-2[24 as our base model to perform the experiments on class-conditional image generation. We use single block of SiT-XL-2 as the Velocity Refiner. We double the input channel from 4 to to take the previous velocity as imput. The resulting velocity refiner only contains 29M parameters, about 4.3% of the original SiT-XL-2(675MM We use ImageNet-1K 16 to train our velocity model. We used Adam [21] optimizer for all models. We constant learning rate of 10-5 and batch size of 18 on single A800 GPU. We used random horizontal flip with probability 0.5 in data augmentation. We did not tune the learning rates, decay/warm-us schedules, AdamW W parameters, or use any extra data augmentation during training. Our velocity refiner (for SiT-XL-2) trains at approximately 4.44 steps/sec on an A800 GPU, and converges in 30,000 steps, which takes about hours.
Text-to-image generation. We use the 2-RF in InstaFlow (0) as our base model to perform the experiments on text-to-image generation. Since the architecture of the original velocity predictor in [20] is U-Net [30], we cannot directly use single block of it as the velocity refiner as we do for SiT 24 Instead, we simply reduce the number of channels in each block from [320, 640, 1280, 1280] to [160, 160, 320, 320] and reduce the number of layers in each block from 2 to 1. We also double the input channel from 4 to to take the previous velocity as input. The resulting velocity refiner only contains 43.5M parameters, about 5% of the original U-Net (860M). We use subset of LAION 34 containing only 50K images to train our velocity model. We use Adam W [2T] optimizer with learning rate of 2e-5 and weight decay of 0.0. We adopt batch size of 16 and set the warming- uup steps as 100. We also use a gradient clipping of 0.01 to stabilize training. We train model on single A800 GPU for 10K iterations, which takes about 5.5 hours.
Implementation of extension tasks. We have demonstrated our FlowTurbo is also suitable for extension tasks due to the multi-step nature of our framework in Section For image inpainting, we adopt the inpainting pipeline in diffusion models 4 where we merge the noise latent and the generated latent specific timestep by the input mask. For object removal, we first use Grounded-SAM to generate the mask and perform similar image inpainting pipeline. For image editing, we adopt SDEdit (25) which first adds noise to the original image and use as an intermediate result to continue the sampling.
D More Analysis
In this section, we provide more analysis through both quantitative results and qualitative results.
2 License: Custom (research, non-commercial)
License: Creative Common CC-BY 4.0
hhaps:/huuggnngfccccoooooo
https://gituub..om/DEEE Reseanch/Grrounde Sewg--eeeeeenettnnneeeeeee
18

---

Table Comparisons with state-of-the-art methods on text-to-image generation. We compare our PlowTurbe with state-of-the-an diffusion models (15 steps
(23) and show our
FlowTurbo enjoys favorable trade-offs between sampling quality and speed.
Method
Sample Config
FLOPs (G)
Latency (ms / img)
FID
SD2.1 30
15 steps DPM++
11427
286.0
33.03
SDXL
20
15 steps DPM+ ++
23
24266
427.2
29.46
PixArt-a
steps DPM
23
17523
366.8
37.96
PixArt-a
3
15 steps DPM++
23
17957
365.9
33.62
FlowTurbo
H1 Rs
4030
104.8
28.60
Turbo
Ps
5386
137.0
27.60
D.1 More Quantitative Results
Ablation of the number of the velocity refiner. In Table 6 we investigate how to choose the number of velocity refiners to get better sampling quality. We adopt two basic configurations of H1R1o and HoRoo: and vary the number of velocity refiners from to We find that the FID will first decrease and then increase when NR becomes larger, and there exists optimal NR 5 where we reach the lowest FID. These results indicate that we can always tune this hyper-parameter to expect better result.
More comparisons on text-to-image generation. In Table Table 7 we compare the sampling quality and speed of FLowTurbo with state-of-the-ard diffusion models on text-to-image generation. For all the diffusion models, we adopt I 15-step DPM-Solver++ (23 as the default sampler. The FLOPs reported also take the multi-step sampling into account. Our results show that FlowTurbo can achieve the lowest FID and inference latency.
D.2 More Qualitative Results
To better illustrate the sampling quality of our Flow'Turbo, we provide more qualitative results both class-conditional image generation and text-to-image generation.
Nass-conditional image generation. We SiT-XL (24 as flow-based model for class- conditional image generation. In Figure 5 we provide random samples from FlowTurbo of the sample config Hg Pg Rs, which inference 100 ms/img. We also demonstrate the sampling quality trade-offs in Figure we compare the sampling quality of two different configurations H1 PrR (38 img) and P9R (100 img). We generate the images from the same initial noise for better comparisons. Our result demonstrates that our FlowTurbo can achieve real-time image generation, and the sampling quality can be further improved with more computational budgets.
Text-to-imant generation. We adopt Lumina-Noe -ieinna--eet--ee Nex 9 to achieve text- sel-to-immae generation. We compare the sampling quality and speed of Heun's 's method and our FlowTurbo in Figure 7 We find that Turborro can consistently generate images with better quality and more visual details, while requiring less inference time.
E
Code
Our code is implemented in PyTorch We use the codebase of [24) to conduct experiments. The code available at htttts:://githuu com/ httmm/himm20/F1oo/rrrlr
"nttps//pytorch.org
19

---

I
CABA
Figure 5: Random samples from FlowTurbo on ImageNet 256 x 256. We use classifier-free guidance scale 4.0 and the sample config of Hg PsRs (100 ms / img)
20

---

(a) Sample Config H, P&P&R Ra (38 ms / img))
(b) Sample Config H3 Rs (100 /img)
Figure 6: Uncurated 256x samples from FlowTurbo (CFG = 4.0). For better visualization. We compare two sample configurations (H₃ PsRs and Hg Rs). The same initial noise used for both sample configurations for better comparisons.
21

---

mugh ini gntheess,, atelding dux1 soes reseahling roons
kitten wearing witch's rate nst, noteing eagie
Undorwater landscage with colorful ecchanical parts, coblo, Ndres
Incide the glass sahere, Pinate Ship In stoms with waves In the Park
flying through
eyes
legendary sinplos
singZe brows bird parchod on mossy branch
Digital votercolo of Samor Scops sunset, With Cowery pasto3 coloes
beautifu) froe insice beautiful beach clirts
silver
Figure 7: More visual comparisons between Heun's method (2.6 / img, lef)) and our FlowTurbe (1.8 s / img, right).
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the main contribution of this paper new framework accelerate the sampling of flow-based generative models, which is fully discussed in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the limitations are discussed in the last paragraph of Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: the theoretical result the convergence of pseudo corrector, which is presented in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we have included implementation details both Section 4.1 and Appendix C We have also included the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: the datasets we used are all publicly available, we have included the code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides/ Soaadd/ss/CoeeSSmmmssisiooooee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccpppllligggdeeeeeeeeeeeee Coaee///ddeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: have included these details in both Section 4.1 and Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
following common practice in 24
we do not need to report error
bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: the compute resources have been discussed in Section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /
Answer: [Yes]
Justification: we have carefully checked the NeurIPS Code of Ethics and make sure our work conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the broader impacts are discussed in the last paragraph of Section 4.4
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: we propose acceleration framework over existing flow-based generative models and thus we do not introduce new risks of misuse. The safeguards needs to be considered in the pre-trained flow-based generative models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all the datasets are cited properly, and the licenses are also provided in Ap- pendix C
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: this paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "jrVHDkK1JS", "reviewer": "Reviewer_yH4R", "summary": "The paper presents a new approach to accelerate the sampling process in flow-based generative models. Unlike diffusion models, flow-based models, which are based on learning velocity fields through flow-matching, have not seen extensive development in efficient sampling techniques. The authors introduce FlowTurbo, a framework that utilizes a lightweight velocity refiner to estimate velocity during sampling, significantly reducing computation time while maintaining or improving image quality. Additionally, the paper proposes pseudo corrector and sample-aware compilation techniques to further enhance sampling efficiency. Experimental results demonstrate that FlowTurbo achieves substantial speedups and high-quality results in both class-conditional and text-to-image generation tasks.", "questions": "My major concern is that the paper considers improving the sampling efficiency of flow-based models which is good, but lacks comparison with other fast-sampling diffusion models. Given the theoretical or practical similarities between diffusion models and flow matching models, I believe the previous methods can be also applied to flow matching models. In that sense, there is a large gap in terms of performance, as previous works can sample within only 2-4 steps, e.g., consistency models. In order to elaborate the author’s claim that the velocity refiner is efficient and as good as those methods, the author should provide some additional comparison with those methods.", "limitations": "The authors mentioned some limitations that those method could not be used for diffusion models, yet it could be in a better version if they demonstrate with detailed analysis.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Technical contribution**: The introduction of a lightweight velocity refiner to approximate the offset between the velocities between adjacent timesteps seems sound and is efficient as it requires only a light-weight model. \n   \n2. **Additional Techniques**: The integration of pseudo corrector and sample-aware compilation techniques shows a well-rounded approach to improving both speed and quality of the generative process.\n\n3. **Extensive Experiments**: The paper provides thorough experimental results, demonstrating the effectiveness of FlowTurbo across different tasks and models, including detailed comparisons with existing methods.\n\n4. **Real-time Performance**: Achieving real-time image generation with substantial improvements in inference time without compromising quality is a notable accomplishment.\n\n5. **Versatility**: FlowTurbo’s compatibility with various applications such as image editing and inpainting highlights its flexibility and potential for broader applications.", "weaknesses": "1. **Limited Comparison**: There are many efficient diffusion distillation methods, e.g., consistency distillation [1,2], consistent trajectory model [3], distribution matching distillation [4], adversarial approaches [5,6] (which also distilled using model trained with rectified flow). However, the paper has not been compared with any of those, which makes the comparison incomplete. In order to describe the technical advantage of using lightweight refiner, the author should compare with full fine-tuning (i.e., distillation) or with parameter efficient fine-tuning, e.g., LoRA. Thus, Table 2 should be filled with various baselines in order to precisely state ‘comparison with the state-of-the-arts’. \n2. **Complexity of implementation**: In addition, while the method seems sound, the method is somewhat complicated to use all Heun’s method, corrector step, etc. This might be an obstacle to scale this method to a large-scale.\n\n**References**\\\n[1] Song, Yang, et al. \"Consistency models.\" arXiv preprint arXiv:2303.01469 (2023). \\\n[2] Song, Yang, and Prafulla Dhariwal. \"Improved techniques for training consistency models.\" arXiv preprint arXiv:2310.14189 (2023).\\\n[3] Kim, Dongjun, et al. \"Consistency trajectory models: Learning probability flow ode trajectory of diffusion.\" arXiv preprint arXiv:2310.02279 (2023).\\\n[4] Yin, Tianwei, et al. \"One-step diffusion with distribution matching distillation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\\\n[5] Sauer, Axel, et al. \"Adversarial diffusion distillation.\" arXiv preprint arXiv:2311.17042 (2023).\\\n[6] Sauer, Axel, et al. \"Fast high-resolution image synthesis with latent adversarial diffusion distillation.\" arXiv preprint arXiv:2403.12015 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About comparisons to distillation-based methods.**\n\n**[Reply]** Thanks for your suggestions. We have detailedly discussed the difference between our FlowTurbo and distillation-based methods in our original paper (L62-L68, L100-L104, L205-L214). Although the previous diffusion distillation methods can be directly adapted to flow-based models, they do not investigate the unique properties of the flow sampling trajectory. Our method, however, aims to offer a training-efficient solution for accelerating flow-based image generation. As Reviewer PyDm suggests, the low additional training costs can make our FlowTurbo have a high impact on the acceleration of generative models. \n\nWe also agree that it would be better to include more distillation methods for comparison in Table 2. We summarize the comparisons between our FlowTurbo and [1-3] on ImageNet in the following table. Note that all these methods only perform experiments on ImageNet 64x64, while our experiment setting follows SiT (ImageNet 256x256) and we cannot find a good flow-based model trained on ImageNet 64x64. The different resolution and base model choices might slightly affect the results. \n\n|Method|Resolution|FID 50K|Latency (ms / img)|Training Costs|GPU|\n|-----|-------|-----|-----|------|----|\n|CD [1]|64x64|4.70| 70.1 | 600K iters x 2048 batch size|64 A100 GPU|\n|iCT-deep [2]|64x64|2.77|70.1 | 800K iters x 4096 batch size| A100 Cluster|\n|CTM [3]| 64x64 | 1.73|70.1 | 30K iters x 2048 batch size|8 A100 GPU|\n|FlowTurbo (ours)| 256x256 |2.22 | 72.5 | 30K iters x 18 batch size| 1 A800 GPU|  \n\nBesides, we also provide comparisons with more distillation methods [4-5,7-8] on text-to-image. We adopt the FID 30K on COCO as the evaluation metric. We do not include [6] since we cannot find the same evaluation metric in their paper and the models have not been released.\n\n|Method|FID 30K |Latency (s)|Training Costs|GPU|\n|-----|-----|-----|------|----|\n|LCM [7]|11.1|0.19| 100K iters x 72 batch size |8 A100 GPU|\n|LCM-LoRA [8]|23.62|0.19| - |-|\n|DMD [4]|14.93|0.09|20K iters x 2304 batch size|72 A100 GPU|\n|ADD-M [5]| 20.33 | 0.09 | 4K iters x 128 batch size| - |\n|FlowTurbo (ours)| 11.95 | 0.14| 10K iters x 16 batch size|1 A800 GPU|\n\nFrom the above results, we demonstrate that our FlowTurbo can achieve competitive sampling quality with previous diffusion distillation methods while requiring much fewer training costs. We will add these results to our revised paper to make the comparison clearer.\n\n[7] Luo, Simian, et al. \"Latent consistency models: Synthesizing high-resolution images with few-step inference.\" arXiv preprint arXiv:2310.04378 (2023).\n\n[8] Luo, Simian, et al. \"LCM-LoRA: A universal stable-diffusion acceleration module.\" arXiv preprint arXiv:2311.05556 (2023).\n\n\n**Q2: About the implementation.**\n\n**[Reply]** Thanks for the valuable comments. The motivation to adopt the velocity refiner is quite clear according to Figure 1, where we show the velocity tends to become more and more stable during the sampling. Heun's method is a well-known ODE sampler that is adopted in our baseline SiT, and the pseudo-corrector is proposed to reduce the inference costs while attaining the same convergence order as Heun's method. Therefore, the sampling paradigm of FlowTurbo is quite intuitive and reasonable. As for the scalability of our method, we have conducted experiments with SiT (675M), InstaFlow (860M), and Lumina-Next-T2I (2B) in our paper. Besides, we also conduct experiments on the provide the newly released Stable-Diffusion-3-Medium (8B) and compare our FlowTurbo with the default sampler Euler's method:\n\n|Method|FID 30K $\\downarrow$|Latency (s / img)|\n|-----|-----|-----|\n|Euler| 28.4 | 2.25 |\n|FlowTurbo| 28.3 | 1.35 | \n\nThe sampling quality is measured by FID 30K $\\downarrow$ on MSCOCO. Besides, we also provide some qualitative comparisons on SD3 and the newly released SOTA model FLUX.1-dev (12B) in Figure A2 and Figure A3 in the attached one-page PDF, where we demonstrate that FlowTurbo can generate more visual details. \n\nWe believe the results on SD3 and FLUX.1-dev, together with the experiments on various flow-based models in our original paper can demonstrate the effectiveness of our FlowTurbo at scale.\n\n**Q3: Applying FlowTurbo for diffusion models.**\n\n**[Reply]** Thanks for pointing out this. In our early experiments, we did try to transfer FlowTurbo to diffusion models. Specifically, we adopted PixArt as the base model (diffusion transformer) and tried to learn a refiner for the $\\epsilon_\\theta$. However, just as Figure 1 suggests, the $\\epsilon_\\theta$ of diffusion models changes a lot during the sampling and it is really hard to regress the offset of $\\epsilon_\\theta$. Alongside attempting to directly regress the offset of $\\epsilon_\\theta$, we have also tried to convert the $\\epsilon_\\theta$ to an equivalent velocity through Equation (32):\n\n$$\\boldsymbol{v}=\\frac{\\dot{\\alpha}_t}{\\alpha_t}\\boldsymbol{x} + (\\dot{\\sigma}_t-\\frac{\\dot{\\alpha}_t\\sigma_t}{\\alpha_t})\\boldsymbol{\\epsilon},$$\nbut the refiner still cannot converge. To better understand the difference between diffusion and flow-based models, we plot the curvature of the sampling trajectories in Figure A1 in the attached PDF, where we show the flow-based models enjoy lower curvature (indicating the sampling path is straighter). We will add more theoretical and empirical analysis in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfZtzeGWz6", "reviewer": "Reviewer_PyDm", "summary": "The authors propose FlowTurbo, a method to adapt pre-trained flow-based generative models for faster sampling. The method is based on the observation that the predicted velocity field is quite stable throughout the integration time and hence at each timestamp only small refinement from the previous step is required. Therefore, the authors propose to train a lightweight refinement network and use it at certain timestamps instead of the original velocity field predictor. In addition to the refinement network the paper also suggests to speed up the Heun's integration method by reusing the velocities predicted at previous corrector steps. All the sampling blocks are compiled for further speed up. As a result, FlowTurbo demonstrates great trade-off between speed and quality compared to prior work and achieves sota results on certain tasks.", "questions": "1) In the paper the order of blocks $H_{N_H}P_{N_P}R_{N_R}$ is fixed, and only the number of the blocks $N_H$, $N_P$ and $N_R$ changes. What is the reason behind this? Have you tried other schemes, e.g. sequentially using multiple blocks of type $H_{N_H}P_{N_P}R_{N_R}$?\n\n2) A rather nitpicky comment: Multiple times throughout the paper it is mentioned that the linear interpolation corresponds to the optimal transport from the noise distribution to the data distribution. It is not precisely correct, as the linear interpolation corresponds to the optimal transport from the noise distribution to the distribution concentrated around a particular data point [14]. Methods that attempt to steer flow-matching towards optimal transport between noise and data involve additional tricks [12, 13].\n\nSome typos:\n1) Line 137: \"interpolation between $x_0$ and $\\cancel{v}$ $\\epsilon$\".\n2) Line 170: \"simulate the $\\cancel{x_t}$  $x_{t_i}$\".\n3) Eq. 15: $\\cancel{x_i \\leftarrow x_{i-1}}$ $x_{t_i} \\leftarrow x_{t_{i - 1}}$", "limitations": "The authors have adequately addressed the limitations and the potential negative societal impact of their work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem of speeding up generative models is of high importance nowadays. While the majority of the methods aims for decreasing the number of denoising steps, the authors of FlowTurbo propose to make each step significantly cheaper. This is achieved at low additional training costs and could have high impact on the methods that explicitly utilise the iterative nature of modern diffusion- or flow-based generative models (such as [1, 2, 3], see references in Weaknesses). The method is well-grounded and carefully ablated. The experiments demonstrate the effectiveness of FlowTurbo compared to default sampling.", "weaknesses": "1) **Prior work**: The paper does a good job with contextualizing it relative to the diffusion-based models. However, references to many flow-based methods are missing, which include:\n\n    - Models that leverage flow matching for generative process [2, 4, 5, 6, 7, 8, 9].\n\n    - And (which is more connected to the topic of the paper) the methods that try to speed up the sampling for flow-based models [10, 11, 12, 13].\n\n2) **Method**: Despite the good results, it seems that it is not trivial how to choose the exact configuration of the sampling blocks, i.e. the order and the number of different steps. From the first sight the configurations presented in the paper are quite random. At least some intuition on how to setup those is required. Otherwise, if the configuration needs to be tuned for every other dataset, the method is not that straight-forward to apply.\n\n3) **Evaluation**: \n    - The tables would be simpler to read if NFEs (number of function evaluations) were also included as a separate column.\n    - Is the step of the baseline method also compiled in tables 1a and 1b? If not, it would be nice to add this experiment, in addition to the ablations, to see the effect of other parts.\n\n[1] Watson et. al, Novel view synthesis with diffusion models, ICLR 2023\n\n[2] Davtyan et. al, Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV 2023\n\n[4] Hu et. al, Motion Flow Matching for Human Motion Synthesis and Editing, arxiv 2023\n\n[5] Fischer et. al, Boosting Latent Diffusion with Flow Matching, arxiv 2023\n\n[6] Davtyan et. al, Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation, AAAI 2024\n\n[7] Davtyan et. al, Enabling Visual Composition and Animation in Unsupervised Video Generation, arxiv 2024\n\n[8] Gui et. al, Depthfm: Fast monocular depth estimation with flow matching, arxiv 2024\n\n[9] Hu et. al, Flow Matching for Conditional Text Generation in a Few Sampling Steps, arxiv 2024\n\n[10] Shaul et. al, Bespoke Solvers for Generative Flow Models, arxiv 2023, ICLR 2024\n\n[11] Lee et. al, Minimizing Trajectory Curvature of ODE-based Generative Models, ICML 2023\n\n[12] Pooladian et. al, Multisample flow matching: Straightening flows with minibatch couplings, ICML 2023.\n\n[13] Tong et. al, Improving and generalizing flow-based generative models with minibatch optimal transport, TMLR 2023.\n\n[14] Lipman et. al, Flow matching for generative modeling, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work! We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About prior works on flow-based methods.**\n\n**[Reply]** Thanks for your suggestions. We agree that these works are important in the area of flow-based models, and we will include the mentioned papers in the revised paper. Specifically, we will modify L31 to:\n\n\n> Alongside the research on diffusion models, flow-based models have garnered increasing attention due to their versatility in modeling data distributions, and have been widely applied in various domains including image generation [5], video generation [6, 7], video prediction [2], human motion synthesis [4], depth estimation [8], text generation [9], etc.\n\nAnd for the acceleration methods [10-13] for flow-based models, we will add the discussion of these methods in our related work (L107):\n\n> There are also some work aim to obtain better flow paths which are straighter and more effective for sampling. These work includes some useful technique such as transformed sampling paths [10], minimizing the trajectory curvature [11], multisample flow matching [12], and minibatch optimal transport [13].\n\n**Q2: About the configuration of the sampling blocks.**\n\n**[Reply]** Sorry for the confusion. According to the observation in Figure 1, the velocity during the sampling would become stable at the final few steps, where we adopted a lightweight refiner to regress the velocity offset. Besides, our pseudo corrector is designed to efficiently achieve 2-order convergence, which requires a 2-order intermediate result as initialization. This explains why we need several Heun's steps at the beginning. We also provide some comparisons of different orders and schemes (including repeating the blocks) as follows: \n\n|Method|FID $\\downarrow$|Latency (ms / img)|\n|-----|----|-----|\n|$H_5 P_7R_3$|2.24|72.4|\n|$P_5 H_7R_3$|2.38|79.2|\n|$P_5 R_7H_3$|13.66|53.7|\n|$H_5 R_7P_3$|30.94|60.4|\n\n|Method|FID $\\downarrow$|Latency (ms / img)|\n|-----|----|-----|\n|$[H_1 P_1R_1]_{\\times 2}$| 8.15 | 34.8 |\n|$H_2P_2R_2$|6.39| 34.8 |\n|$[H_1 P_1R_1]_{\\times 3}$| 4.34 | 45.4 |\n|$H_3P_3R_3$|3.34| 45.4 |\n\nAs for the number of different steps, we have included some comparisons in Table 3 and Table 4. Generally speaking, we choose to apply the refiner at the last $20\\sim 40$% sampling steps where the velocity is relatively stable, and we can always adjust the ratio between Heun's steps and pseudo-corrector steps to control the trade-offs between sampling quality and complexity (Table 3d). Besides, since adjusting the configuration does not require further training, we can always select a good configuration according to the computational budgets and qualitative results. We will add these detailed analyses and discussion to our revised paper to provide better guidelines on how to design the configuration of our FlowTurbo.\n\n**Q3: About the evaluation.**\n\n**[Reply]** Thanks for the comments. We provide the results in Table 1a and 1b with a column of NFEs and the baseline methods compiled as follows. We use $f_1$ to denote the function evaluation of the original velocity predictor and $f_2$ to denote the evaluation of our velocity refiner.\n\n**_Table 1a: Class-conditional Image Generation, ImageNet (256x256)_**\n| Method        | Sample Config    | FLOPs (G) | FID 50K $\\downarrow$ | Latency (ms / img)   | NFE  |\n|---------------|------------------|-----------|------|-----------------------|------|\n| Heun's        | $H_8$            | 1898      | 3.68 | 68.0                  |  16 $f_1$  |\n| FlowTurbo     | $H_2 P_4 R_2$    | 957       | 3.63 | 41.6       |     8 $f_1$ + 2 $f_2$ |\n| Heun's        | $H_{11}$         | 2610      | 2.79 | 88.2                 |   22 $f_1$ |\n| FlowTurbo     | $H_2 P_8 R_2$    | 1431      | 2.69 | 55.2          |  12 $f_1$ + 2 $f_2$    |\n| Heun's        | $H_{15}$         | 3559      | 2.42 | 115.3                 | 30 $f_1$    |\n| FlowTurbo     | $H_5 P_7 R_3$    | 2274      | 2.22 | 72.5        |   17 $f_1$ + 3 $f_2$   |\n| Heun's        | $H_{24}$         | 5694      | 2.20 | 176.2                 | 48 $f_1$    |\n| FlowTurbo     | $H_8 P_9 R_5$    | 3457      | 2.12 | 100.3       |   25 $f_1$ + 5 $f_2$   |\n\n**_Table 1b: Text-to-image Generation, MS COCO 2017 (512x512)_**\n| Method        | Sample Config    | FLOPs (G) | FID 5K $\\downarrow$ | Latency (ms / img)   | NFE  |\n|---------------|------------------|-----------|------|-----------------------|------|\n| Heun's        | $H_4$            | 3955      | 32.77 | 92.0                |   8 $f_1$   |\n| FlowTurbo     | $H_1 P_2 R_2$    | 2649      | 32.48 | 68.4        |   4 $f_1$ + 2 $f_2$    |\n| Heun's        | $H_5$            | 4633      | 30.73 | 108.0                |  10 $f_1$   |\n| FlowTurbo     | $H_1 P_4 R_2$    | 3327      | 30.19 | 84.5      |  6 $f_1$ + 2 $f_2$    |\n| Heun's        | $H_8$            | 6667      | 28.61 | 156.2                |  16 $f_1$    |\n| FlowTurbo     | $H_1 P_6 R_3$    | 4030      | 28.60 | 104.8      |  8 $f_1$ + 3 $f_2$    |\n| Heun's        | $H_{10}$         | 8023      | 28.06 | 188.4                |   20 $f_1$   |\n| FlowTurbo     | $H_3 P_6 R_3$    | 5386      | 27.60 | 137.0       |   12 $f_1$ + 3 $f_2$   |\n\n\nThese results clearly show that our method still surpasses the compiled baseline by large margins, demonstrating the effect of our velocity refiner and the pseudo corrector. We will modify these in the revised paper.\n\n**Q4: About the description of the linear interpolation.**\n\n**[Reply]** Thanks for pointing out this. We agree that linear interpolation connects noise distribution to the distribution concentrated around a particular data point in the conditional flow-matching defined in [14]. We will modify the description in L37, L89, L141 in the revision.\n\n**Q5: About the typos.**\n\n**[Reply]** Thanks for your careful reading. We will modify these typos accordingly in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhJhBvOYVq", "reviewer": "Reviewer_RYd7", "summary": "This paper explores enhancing flow-based generative models for image generation by accelerating the sampling process while maintaining or improving image quality. The key contribution, FlowTurbo, is a new framework that introduces a lightweight velocity refiner to adjust velocity predictions during sampling, reducing computational costs. Additionally, techniques like pseudo correctors and sample-aware compilation further optimize the sampling speed. The results show significant acceleration in image generation tasks with improved FID scores, establishing a new benchmark for real-time image generation.", "questions": "If I understand correctly, the proposed method can be applied to diffusion models as well. (Although Figure 1 illustrates flow models may be more suitable.) Have you tested the results on diffusion models like SD or PixArt?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. This paper starts from the distinct features of flow models and naturally proposes corresponding solutions. Comprehensive ablations validate the effectiveness of the proposed methods.", "weaknesses": "- The design of three types of blocks in sampling is somewhat confusing to me. Why do you have to use these types of blocks and arrange them in this order? Do you have to empirically decide the order and numbers of these blocks for each task/domain/model?\n- The visualization in Figure 1 is interesting. However, visualizing the discretization error or curvature [1] during sampling seems to be a more suitable and accurate method.\n- Some important related works [1,2], tailored for flow models, should be discussed and compared.\n\n[1] Nguyen, Bao, Binh Nguyen, and Viet Anh Nguyen. \"Bellman optimal step-size straightening of flow-matching models.\" *arXiv preprint arXiv:2312.16414* (2023).\n\n[2] Shaul, Neta, et al. \"Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models.\" *arXiv preprint arXiv:2403.01329* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work! We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About the design of sampling blocks.**\n\n**[Reply]** Sorry for the confusion. As illustrated in Figure 1, the velocity during the sampling process stabilizes in the final few steps. At this stage, we employ a lightweight refiner to adjust the velocity offset. Additionally, our pseudo corrector is crafted to efficiently achieve second-order convergence, which requires second-order intermediate results (achieved by several Heun's steps at the start) for initialization. Consequently, the sequence of the three block types should be: Heun's block, pseudo corrector block, and refiner block. As the results shown below, changing the order of the blocks will lead to worse sampling quality:\n\n|Method|FID $\\downarrow$|Latency (ms / img)|\n|-----|----|-----|\n|$H_5 P_7R_3$|2.24|72.4|\n|$P_5 H_7R_3$|2.38|79.2|\n|$P_5 R_7H_3$|13.66|53.7|\n|$H_5 R_7P_3$|30.94|60.4|\n\nRegarding the number of steps for each block, we have included comparisons in Table 3 and Table 4. Generally, we apply the refiner during the last 20-40% of the sampling steps, where the velocity is relatively stable. We can always adjust the ratio between Heun's steps and pseudo-corrector steps to balance sampling quality and complexity (as seen in Table 3d). Moreover, since reconfiguring the steps doesn't require additional training, we can select an optimal configuration based on computational resources and qualitative results. We will incorporate this detailed analysis and discussion into our revised paper to offer better guidance on designing the configuration of our FlowTurbo.\n\n\n**Q2: About the visualization.**\n\n**[Reply]** Many thanks for your valuable advice. We agree that visualizing the curvature is more suitable and accurate. We provide the comparisons of the curvatures of different models including DiT, SiT, SD3, FLUX.1-dev (a newly released SOTA flow model), Open-Sora (a flow-based video generation model) in the Figure A1 in the attached PDF. The visualization of curvature basically aligns with our original PCA visualization, where we find the flow-based models enjoys much smaller curvature, indicating the sampling trajectory is straighter. We will replace the visualization in Figure 1 with curvature in our revised paper.\n\n**Q3: About the related work.**\n\n**[Reply]** Thanks for pointing out this. We agree that these two papers are related to our paper and we will discuss them in the related work (L107) as follows:\n\n> There are also some works to accelerate the sampling of flow-based models, such as computing the optimal sampling stepsizes [1] and solver distillation approaches [2].\n\n\nIn our FlowTurbo, we do not consider the variation of stepsizes during the sampling, which makes our method orthogonal to [1]. In other words, we can combine [1] and our FlowTurbo to improve the sampling efficiency. Following [1], we perform experiments on CelebA-HQ and the results are as follows:\n\n|Method|NFE|FID$\\downarrow$|\n|--|---|---|\n|Euler|6|127.01|\n|Bellman [1]|6| 72.54| \n|Bellman [1] + FlowTurbo (ours)|6|**68.74** |\n|Euler|8|109.42|\n|Bellman [1]|8| 49.80| \n|Bellman [1] + FlowTurbo (ours)|8|**45.95** |\n\nIn the following table, we compare our FlowTurbo with Bespoke Non-Stationary (BNS) Solvers [2] on ImageNet. Note that BNS only provides the results on ImageNet 64x64 while we adopt the experimental while our experiment setting follows SiT (ImageNet 256x256) and we cannot find a good publicly available flow-based model trained on ImageNet 64x64. The different resolution and base models choices might slightly affect the results. We also report the number of function evaluations (NFE) in the table, where we use $f_1$ and $f_2$ ($f_2\\approx 0.1 f_1$) to represent the velocity predictor and refiner, respectively.\n\n|Method|Resolution|ImageNet FID 50K| NFE |Training Costs|\n|-----|-------|-----|-----|------|\n|BNS| 64x64| 2.62|16 | 2-10 GPU days|\n|FlowTurbo ($H_5 P_5 R_2$)| 256x256 |2.46 | 15 $f_1$ + 2 $f_2$ | 2 GPU hours|\n\nWe also provide the result on MS COCO text-to-image generation:\n|Method|Resolution|COCO FID 30K| NFE |Training Costs|\n|-----|-------|-----|-----|-----|\n|BNS| 512x512|14.68| 20 | 15-24 GPU days|\n|FlowTurbo ($H_3 P_6 R_3$)| 512x512|11.95 | 12 $f_1$ + 3 $f_2$ | 5.5 GPU hours|\n\nThese results clearly show that our FlowTurbo is better in both sampling quality and training efficiency. Besides, BNS solvers struggle to generalize unseen NFEs while our FlowTurbo allows flexible adjustment of the sampling configuration after the velocity has been trained. We will add the above discussion and comparisons in our revised paper.\n\n**Q4: Applying FlowTurbo for diffusion models.**\n\n**[Reply]** That's a good question. In our early experiments, we did try to transfer a similar idea of FlowTurbo to diffusion models. Specifically, we adopted PixArt as the base model (diffusion transformer) and tried to learn a refiner for the $\\epsilon_\\theta$. However, just as Figure 1 suggests, the $\\epsilon_\\theta$ of diffusion models changes a lot during the sampling and it is really hard to regress the offset of $\\epsilon_\\theta$. We have also tried to convert the $\\epsilon_\\theta$ to an equivalent velocity through Equation (32):\n\n$$\\boldsymbol{v}=\\frac{\\dot{\\alpha}_t}{\\alpha_t}x + (\\dot{\\sigma}_t-\\frac{\\dot{\\alpha}_t\\sigma_t}{\\alpha_t})\\boldsymbol{\\epsilon},$$\nbut the refiner still cannot converge. We think some techniques such like [3] to transform the sampling path to be straighter could be helpful, and we will leave this for future work. \n\n[3] Shaul et. al, Bespoke Solvers for Generative Flow Models, arxiv 2023, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zT0StyzJP4", "reviewer": "Reviewer_Q4Pt", "summary": "This paper introduces FlowTurbo, a framework to accelerate flow-based generative models. The key contributions are:\n1) a lightweight velocity refiner to estimate the offset of velocity efficiently during sampling;\n2) a pseudo corrector to reduce the number of model evaluations while keeping the original second-order convergence;\n3) sample aware compilation to compile model evaluations, sampling steps and cfg into a static graph for extra speedup;\n\nExtensive experiments with both class-conditioned and text-to-image generation show significant acceleration (30-58%). Notably, on ImageNet 256x256 generations, FlowTurbo achieves the fastest sampling speed (38 ms/image) with FID 3.93. Moreover, due to the compatibility with multi-step sampling, it can also be applied to other tasks including image editing, inpainting, etc.", "questions": "Can this method also be applied to video-based generation models? The inference is much slower for video generation compared with image generation. it would be great to see if there is a unified method for both image and video generation.", "limitations": "The proposed method is limited to flow-based generation models.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Given the popularity of flow-based generation models, SD3 for image and Sora for video, flow-based model acceleration is an important area, which is under-explored compared with diffusion-based models. \n\nMotivated by the stability of velocity prediction during sampling, the authors proposed a lightweight velocity refiner. The pseudo corrector technique reduces the model evaluation while preserving the second-order convergency.  These ideas are novel and effective. The experimental results are comprehensive covering generation task with class-conditional and text-to-image generation, editing tasks including editing, inpainting, object removal. The ablation studies are thorough, examining each component's contribution and various hyperparameters. Finally, the code is also provided for reproducibility. The paper is easy to follow, for example, Figure 2 illustrates the motivation for light-weight estimation model very clearly.", "weaknesses": "For text-to-image generation, insta-flow is adopted as the teacher model, which is built on top of sd1.5. It would be great to experiment with other flow-based models that are natively trained with flow matching, for example, sd3. \nIt would also be interesting to see how FlowTurbo performs when the model size changes. For example, SiT-XL vs SiT-L. \nFinally, one missing ablation is the architecture of refiner. how to design a refiner for different architectures?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work! We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About experiments on other models.**\n\n**[Reply]** Thanks for your advice. Since SD3 is open-sourced a few months later than the submission, we didn't include experimental results on it in our original paper. In the following table, we compare the sampling quality (measured by FID 30K $\\downarrow$ on MSCOCO) and the sampling speed of the default sampler of SD3 (Euler's method) and our FlowTurbo: \n\n| Method    | FID 30K $\\downarrow$ | Latency (s / img) |\n| --------- | -------------------- | ----------------- |\n| Euler     | 28.4                 | 2.25              |\n| FlowTurbo | 28.3                 | 1.35              |\n\nBesides, we also provide qualitative comparisons on SD3 and the newly released SOTA model FLUX.1-dev in Figure A2 and Figure A3 in the attached one-page PDF, where we demonstrate FlowTurbo can generate more details than the baseline.\n\n**Q2: About different model sizes.**\n\n**[Reply]** Thanks for pointing out this. In our original paper, we conducted experiments with models of various sizes including SiT (675M), InstaFlow (860M), Lumina-Next-T2I (2B). We also agree that it is interesting to see the performance on similar architectures with different model sizes such as SiT-XL and SiT-L. However, SiT only released their SiT-XL-2 model and we failed to find another publicly available family of models that satisfied the requirement. We will add the comparison once the SiT releases another model such as SiT-L or SiT-B.\n\n**Q3. About the refiner architecture**\n\n**[Reply]** When designing the architecture for the velocity refiner, we followed a simple rule to make the refiner have a similar architecture as the base velocity predictor but with much fewer parameters (~5% of the base model). The detailed architecture is described in Section 4.1 and Appendix C. For example, since SiT consists of multiple transformer blocks, we simply use a single block as the refiner. For text-to-image generation, we reduce the number of layers and channels of the UNet. In our early experiments, we did have tried another architecture for class-conditional image generation, where a SiT-S (a smaller version of the base velocity predictor SiT-XL) is adopted as the refiner (as shown in the following table):\n\n| Sample Config | Refiner Architecture | Params | FID  |\n| ------------- | -------------------- | ------ | ---- |\n| $H_5 P_7 R_3$ | SiT-S                | 33M    | 2.53 |\n| $H_5 P_7 R_3$ | a block of SiT-XL    | 29M    | 2.22 |\n| $H_8 P_9 R_5$ | SiT-S                | 33M    | 2.24 |\n| $H_8 P_9 R_5$ | a block of SiT-XL    | 29M    | 2.12 |\n\nWe find that using a block of SiT-XL as the refiner is slightly better than the SiT-S. These results demonstrate that our framework is robust to the choice of model architectures for the velocity refiner. We will add these discussions and results to Appendix C of the revised paper.\n\n\n**Q4. About the application to video generation models**\n\n**[Reply]** Thanks for your advice. Since our FlowTurbo is designed to accelerate flow-based generative models, it can be applied to video generation models as long as they are trained via flow-matching (i.e., has a stable velocity during the sampling). Specifically, we experiment with the recently released model Open-Sora and provide the qualitative results in Figure A4 of the attached one-page PDF, where we show our FlowTurbo generates video with higher fidelity under the same computational budgets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for the positive feedback and valuable comments on our work. In the attached one-page PDF, we provide more visualizations and qualitative results to better illustrate the motivation and effectiveness of our FlowTurbo. In Figure A1, we compare the curvature (as suggested by Reviewer RYd7) of the sampling trajectories of different models including DiT, SiT, SD3-Medium, FLUX.1-dev, and Open-Sora. In Figure A2-A4, we provide more qualitative comparisons on SD3-Medium (a SOTA flow model by StabilityAI, released on June 12th), FLUX.1-dev (a SOTA flow model by Black Forest Labs, released on August 3rd), Open-Sora (a video-generation flow model). These qualitative clearly demonstrate our FlowTurbo generates visual content with higher fidelity and more details than the default sampler.

Meta-review: The paper proposes several improvements to sampling of flow-based models, including lightweight velocity refiner and "pseudo corrector". These together improve the quality-speed tradeoff of sampling, resulting in SOTA quality/speed frontier.

The paper is clearly written, the proposed modifications make sense, the experiments are thorough, and the results are good. All reviewers recommend acceptance. I concur and recommend acceptance too.
Decision: Accept (poster)