Title: FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner
Venue: NeurIPS 2024
Accepted: yes
Keywords: flow-based generation

diffusion models

efficient sampling

image generation
Abstract: Building on the success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc. By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1\%$\sim$58.3\% on class-conditional generation and 29.8\%$\sim$38.5\% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-art. Code is available at https://github.com/shiml20/FlowTurbo.

FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner

Abstract

Building on the success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc . By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1% ∼ 58.3% on class-conditional generation and 29.8% ∼ 38.5% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-art. Code is available at https://github.com/shiml20/FlowTurbo .

1 Introduction

In recent years, diffusion models have emerged as powerful generative models, drawing considerable interest and demonstrating remarkable performance across various domains[10, 38, 30, 12]. Diffusion models utilize a denoising network, ϵ θ , to learn the reverse of a diffusion process that gradually adds noise to transform the data distribution into a Gaussian distribution. While the formulation of diffusion models enables stable training and flexible condition injection[30], sampling from these models requires iterative denoising. This process necessitates multiple evaluations of the denoising network, thereby increasing computational costs. To address this, several techniques such as fast

diffusion samplers[22, 18, 42] and efficient distillation[31, 37] have been proposed to reduce the sampling steps of diffusion models.

Alongside the research on diffusion models, flow-based models[5, 19, 17] have garnered increasing attention due to their versatility in modeling data distributions. Flow is defined as a probability path that connects two distributions and can be efficiently modeled by learning a neural network to estimate the conditional velocity field through a neural network v θ via flow matching [17]. Encompassing the standard diffusion process as a special case, flow-based generative models support more flexible choices of probability paths. Recent work has favored a simple linear interpolant path [20, 24, 8], which corresponds to the optimal transport from the Gaussian distribution to the data distribution. This linear connection between data and noise results in a more efficient sampling process for flowbased models. However, unlike diffusion models, which benefit from numerous efficient sampling methods, current samplers for flow-based models primarily rely on traditional numerical methods such as Euler's method and Heun's method [24]. These traditional methods, while functional, fail to fully exploit the unique properties of flow-based generative models, thereby limiting the potential for faster and more efficient sampling.

In this paper, we propose FlowTurbo, a framework designed to accelerate the generation process of flow-based generative models. FlowTurbo is motivated by comparing the training objectives of diffusion and flow-based generative models, as well as analyzing how the prediction results ϵ θ and v θ vary over time. Our observation, illustrated in Figure 1, indicates that the velocity predictions of a flow-based model remain relatively stable during sampling, in contrast to the more variable predictions of ϵ θ in diffusion models. This stability allows us to regress the offset of the velocity at each sampling step using a lightweight velocity refiner, which contains only 5% of the parameters of the original velocity prediction model. During the sampling process, we can replace the original velocity prediction model with our lightweight refiner at specific steps to reduce computational costs.

As a step towards real-time image generation, we propose two useful techniques called pseudo corrector and sample-aware compilation to further improve the sampling speed. Specifically, the pseudo corrector method modifies the updating rule in Heun's method by reusing the velocity prediction of the previous sampling step, which will reduce the number of model evaluations at each step by half while keeping the original convergence order. The sample-aware compilation integrates the model evaluations, the sampling steps as well as the classifier-free guidance [11] together and compile them into a static graph, which can bring extra speedup compared with standard model-level compilation. Since each sample block is independent, we can still adjust the number of inference steps and sampling configurations flexibly.

Our FlowTurbo framework is fundamentally different from previous one-step distillation methods for diffusion models [20, 40, 32], which require generating millions of noise-image pairs offline and conducting distillation over hundreds of GPU days. In contrast, FlowTurbo's velocity refiner can be efficiently trained on pure images in less than 6 hours. Moreover, one-step distillation-based methods are limited to image generation and disable most of the functionalities of the original base model. Conversely, FlowTurbo preserves the multi-step sampling paradigm, allowing it to be effectively applied to various tasks such as image editing, inpainting, and more.

We perform extensive experiments to evaluate our method. By applying FlowTurbo to different flow-based models, we obtain an acceleration ratio of 53.1% ∼ 58.3% on class-conditional generation and 29.8% ∼ 38.5% on text-to-image generation. Notably, FlowTurbo attains an FID score of 2.12 on ImageNet with 100 (ms / img) and FID of score 3.93 with 38 (ms / img), thereby enabling realtime image generation and establishes the new state-of-the-art. Additionally, we present qualitative comparisons demonstrating how FlowTurbo generates superior images with higher throughput and how it can be seamlessly integrated into various applications such as image editing, inpainting, etc . We believe our FlowTurbo can serve as a general framework to accelerate flow-based generative models and will see wider use as these models continue to grow [24, 20, 8, 9].

2 Related Work

Diffusion and flow-based models. Diffusion models [10, 38] are a family of generative models that have become the de-facto method for high-quality generation. The diffusion process gradually adds noise to transform the data distribution to a normal distribution, and the goal of diffusion models is to use a network ϵ θ to learn the reverse of the diffusion process via score-matching [10, 38]. Rombach et

4

8

12

16

20

Sampling Steps

0.00

0.18

0.35

0.53

0.70

Normalized Value of Curvature

SiT

DiT

SD3-Medium

FLUX.1-dev

Open-Sora

(a) Comparison of curvatures of different models.

4

8

12

16

20

0.00

0.01

0.02

0.03

SiT

(b) SiT

4

8

12

16

20

0.00

0.01

0.02

0.03

SD3-Medium

(c) SD3-Medium

4

8

12

16

20

0.00

0.01

0.02

0.03

FLUX.1-dev

(d) FLUX.1-dev

4

8

12

16

20

0.00

0.01

0.02

0.03

Open-Sora

(e) Open-Sora

al . [30] first scales up diffusion models to large-scale text-to-image generation by performing the diffusion on latent space and adopting cross-attention to inject conditions. The pre-trained diffusion models can also be easily fine-tuned to achieve generation with more diverse conditions [41, 27] and have attracted increasing attention in the community. Flow-based generative models are different from diffusion models in both data modeling and training objectives. Flow-based models [20, 17, 8, 24] consider the probability path from one distribution to another, and learn the velocity field via flow matching [17]. By choosing the linear interpolant as the probability path which corresponds to the optimal transport from the normal distribution to the data distribution, the trajectory from noise to data becomes more straighter which is beneficial to the sampling. Recent work [24, 8] have demonstrates the effectiveness and scalability of flow-based generation models. However, both diffusion and flow-based models requires multiple evaluations of the prediction model, leading to lower inference speed than traditional architectures like GAN. In this work, we focus on this issue and aim to accelerate flow-based generative models.

Efficient visual generation. Accelerating the generation of diffusion models has become an increasingly important topic. Existing methods can be roughly categorized as training-free and training-based methods. Training-free methods aim to design faster samplers that can reduce the approximation error when sampling from the diffusion SDE or ODE [36, 22, 18, 42], while keeping the weights of diffusion models unchanged. Training-based methods often aim to reshape the sampling trajectory by distillation from the diffusion model [31, 40] to achieve the few-step or even one-step generation. These training-based methods usually requires multiple-round of distillation [31, 20] and expensive training resources ( e.g ., > 100 GPU days in [20]). Besides, the distilled one-step model no longer supports image editing due to the lack of multi-step sampling. Although there are a variety of methods for accelerating diffusion models, there are few fast sampling methods designed for flow-based generative models. Existing flow-based models adopt traditional numerical methods like Euler's method or Heun's method during the inference [24]. In this work, we provide a framework called FlowTurbo to accelerate the generation of flow-based models by learning a lightweight velocity refiner (which only requires <6 GPU hours) to regress the offset of the velocity. Together with other proposed techniques, FlowTurbo addresses the previously unmet need for an efficient flow-based generation framework, paving the way for real-time generative applications.

3 Method

3.1 Preliminaries: Diffusion and Flow-based Models

Diffusion models. Recently, diffusion models [10, 38, 35, 30] have emerged as a powerful family of generative models. The diffusion models are trained to learn the inverse of a diffusion process such

that it can recover the data distribution p 0 ( x 0 ) from the Gaussian noise. The diffusion process can be represented as:

where α t , σ t are the chosen noise schedule such that the marginal distribution p 1 ( x 1 ) ∼ N (0 , I ) . The optimization of diffusion models can be derived by either minimizing the ELBO of the reverse process [10] or solving the reverse diffusion SDE [38], which would both lead to the same training objective of score-matching, i.e ., to learn a noise prediction model ϵ θ ( x t , t ) to estimate the scaled score function -σ t ∇ x log p t ( x t ) :

where λ ( t ) is a time-dependent coefficient. Sampling from a diffusion model can be achieved by solving the reverse-time SDE or the corresponding diffusion ODEs [38], which can be efficiently achieved by modern fast diffusion samplers [36, 22, 42].

Flow-based models. Flow-based models can be traced back to Continuous Normalizing Flows [5] (CNF), which is a more generic modeling technique and can capture the probability paths of the diffusion process as well [17]. Training a CNF becomes more practical since the purpose of the flow matching technique [17], which learns the conditional velocity field of the flow. Similar to (1), we can add some constraints to the noise schedule such that α 0 = 1 , σ 0 = 0 and α 1 = 0 , σ 1 = 1 , and then define the flow as:

In this case, the velocity field that generates the flow ψ t can be represented as:

The training objective of conditional flow matching is to train a velocity prediction model v θ to estimate the conditional velocity field:

The sampling of a flow-based model can be achieved by solving the probability flow ODE with the learned velocity

Since the formulation of the flow ψ t can be viewed as the interpolation between x 0 and v , it is also referred to as interpolant in some literature [1, 24]. Among various types of interpolants, a very simple choice is linear interpolant [24, 8], where α t = (1 -t ) and σ t = t . In this case, the velocity field becomes a straight line connecting the initial noise and the data point, which also corresponds to the optimal transport between the two distributions [19, 17]. The effectiveness and scalability of the linear interpolant have also been proven in recent work [17, 24, 8].

3.2 Efficient Estimation of Velocity

We consider the velocity estimation in flow-based generative models with the linear interpolant [24, 8, 20]. As shown in (5), the training target of the velocity prediction model v θ is exactly ϵ -x 0 , a constant value independent of t . Our main motivation is to efficiently estimate the velocity during the sampling, instead of evaluating the whole velocity prediction model v θ every time.

Analyzing the stability of velocity. We start by analyzing the stability of the output value of v θ along the sampling trajectory. By comparing the training objectives of diffusion and flow-based models (2)(5), we know that the target of v θ is independent of t . A more in-depth discussion is provided in Appendix A.3, where we show the two training objectives have different time-dependent weight functions. To verify whether there are similar patterns during the sampling, we compare how the prediction results change across the sampling steps in Figure 1. Specifically, we compare the curvatures of ϵ θ of a diffusion model (DiT [28]) and the v θ of flow-based models (SiT [8], SD3 [8], etc) during the sampling steps. For each model, we sample from 8 random noises and set the total sampling steps as 20. It can be clearly observed that v θ of a flow-based model is much more stable

+

𝐱𝑡 𝑖-1

= 𝜓𝑡𝑖-1

𝐱0

𝝐

= 1 - 𝑡𝑖-1

𝐱0 +𝑡𝑖-1𝝐

𝐱0

∼ 𝑝0

𝐱 ,𝝐 ∼ 𝒩(0,𝐈)

𝐱𝑡 𝑖-1

Velocity

Predictor

𝐯𝜃

𝐯𝑡 𝑖-1

× Δ𝑡

Velocity

Refiner

𝐫 𝜙

𝐯𝑡 𝑖-1

෤

+

ℒ𝜙 = 𝔼

෤

-𝐯𝜃(𝐱𝑡𝑖

, 𝑡

𝑖

)

2

2

×Δ𝑡

+

𝐱𝑡 𝑖-1

Velocity

Predictor

𝐯𝜃

𝐯𝑡 𝑖-1

× Δ𝑡

Velocity

Predictor

𝐯𝜃

ത

ത

+

×0.5

+

(b)

Heun's Method Sample Block

(a)

Learning a Lightweight V elocity Refiner

×Δ𝑡

+

𝐱𝑡 𝑖-1

Velocity

Cache

ത

𝐯𝑡 𝑖-1

× Δ𝑡

Velocity

Predictor

𝐯𝜃

ത

ത

+

×0.5

+

𝐱𝑡0

∼ 𝒩(0,𝐈)

(c)

(d)

FlowTurbo Sampling

𝐱𝑡0

SAC

Heun's Method

Sample Block

SAC

Pseudo Corrector

Sample Block

SAC

Sample block

𝐱𝑡𝑁

SAC

: Sample-A ware  Compilation

× 𝑁𝐻

× 𝑁𝑃

× 𝑁𝑅

𝑁 = 𝑁𝐻 + 𝑁𝑃 + 𝑁𝑅

than the ϵ θ of a diffusion model. Therefore, We define the v θ as a 'stable value' . The stability of v θ makes it possible to obtain the velocity more efficiently rather than performing the forward pass of the whole velocity prediction network v θ at every sampling step.

Learning a lightweight velocity refiner. Since the velocity in a flow-based model is a 'stable value', we propose to learn a lightweight refiner that can adjust the velocity with minimal computational costs. The velocity refiner takes as inputs both the current intermediate result and the velocity of the previous step, and returns the offset of velocity:

The velocity refiner r ϕ can be designed to be very lightweight ( < 5% parameters of v θ ). The detailed architecture can be found in Appendix C.

To learn the velocity refiner, we need to minimize the difference between the output of r ϕ and the actual offset v t i -v t i -1 . However, it requires multiple-step sampling to obtain an intermediate result x t i to make the training objective perfectly align with our target. To reduce the training cost, we simulate the x t with one-step sampling starting from x t i -1 , which is directly obtained by the flow ψ t i -1 . The detailed procedure to compute the loss is listed as follows:

Where ∆ t = t i -t i -1 and we use a simple Euler step for the Solver to obtain x t i . Once the velocity refiner is learned, we can use it to replace the original v θ at some specific sampling steps. We will demonstrate through experiments that adding the velocity refiner can improve the sampling quality without introducing noticeable computational overhead.

Compatibility with classifier-free guidance. Classifier-free guidance [11] is a useful technique to improve the sampling quality in conditional sampling. Let y be the condition, the classifier-free guidance for a velocity prediction model [8] can be defined as:

where ζ is the guidance scale and ∅ denotes the null condition. To make our velocity refiner support classifier-free guidance, we only need to make sure both the conditional prediction v θ ( x , t | y ) and the

unconditional prediction v θ ( x , t | ∅ ) appear during the training. Note that we always feed the velocity prediction model v θ and the velocity refiner r ϕ with the same condition.

where we set the γ 1 = 0 . 1 as the probability of using an unconditional velocity.

3.3 Towards Real-Time Image Generation

The sampling costs of a flow-based model can be significantly minimized by integrating our lightweight velocity refiner r ϕ in place of the velocity prediction network v θ at selected sampling steps. In this section, we propose two techniques to further improve the sampling speed towards real-time image generation.

Pseudo corrector. Traditional numerical ODE solvers are usually used to sample from a probability flow ODE. For example, SiT [8] adopt a Heun method (or improved Euler's method) [15] as the ODE solver. The update rule from t i -1 to t i can be written as:

Each Heun step contains a predictor step (14) and a corrector step (15), thus includes two evaluations of the velocity predictor v θ , bringing extra inference costs. Motivated by [42], we propose to reuse the d i in the next sampling step, instead of re-computing it via d i ← v θ ( x t i , t i | y ) (see Figure 2 (b)(c) for illustration). We call this a pseudo corrector since it is different from the predictor-corrector solvers in numerical analysis. It can be proved (see Appendix B) that the pseudo corrector also enjoys 2-order convergence while only having one model evaluation at each step.

Sample-aware compilation. Compiling the network into a static graph is a widely used technique for model acceleration. However, all the previous work only considers network-level compilation, i.e ., only compiling the ϵ θ or v θ . We propose the sample-aware compilation which wraps both the forward pass of v θ or r ϕ and the sampling operation together (including the classifier-free guidance) and performs the compilation. For example, the sample blocks illustrated in Figure 2 (b, c) are compiled into static graphs. Since each sample block is independent, we can still adjust the number of inference steps and sampling configurations flexibly.

3.4 Discussion

Recently, there have been more and more training-based methods [20, 40, 32] aiming to accelerate diffusion models or flow-based models through one-step distillation. Although these methods can achieve faster inference, they usually require generating paired data using the pre-trained model and suffer from large training costs ( e.g ., >100 GPU days in [40, 20]). Besides, one-step methods only keep the generation ability of the original model while disabling more diverse applications such as image inpainting and image editing. In contrast, our FlowTurbo aims to accelerate flow-based models through velocity refinement, which still works in a multi-step manner and performs sampling on the original trajectory. For example, FlowTurbo can be easily combined with existing diffusion-based image editing methods like SDEdit [25] (see Section 4.4).

4 Experiments

We conduct extensive experiments to verify the effectiveness of FlowTurbo. Specifically, we apply FlowTurbo to both class-conditional image generation and text-to-image generation tasks and demonstrate that FlowTurbo can significantly reduce the sampling costs of the flow-based generative models. We also provide a detailed analysis of each component of FlowTurbo, as well as qualitative comparisons of different tasks.

4.1 Setups

In our experiments, we consider two widely used benchmarks including class-conditional image generation and text-to-image generation. For class-conditional image generation, we adopt a transformer-

(a) Class-conditional Image Generation

(b) Text-to-image Generation

style flow-based model SiT-XL [24] pre-trained on ImageNet 256 × 256. For text-to-image generation, we utilize InstaFlow [20] as the flow-based model, whose backbone is a U-Net similar to StableDiffusion [30]. Note that we use the 2-RF model from [19] instead of the distilled version since our FlowTurbo is designed to achieve acceleration within the multi-step sampling framework. The velocity refiner only contains 4.3% and 5% parameters of the corresponding predictor, and the detailed architecture can be found in Appendix C. During training, we randomly sample ∆ t ∈ (0 , 0 . 12] and compute the training objectives in (13). In both tasks, we use a single NVIDIA A800 GPU to train the velocity refiner and find it converges within 6 hours. We use a batch size of 8 on a single A800 GPU to measure the latency of each method. Please refer to Appendix C for more details.

4.2 Main Results

Class-conditional image generation. Weadopt the SiT-XL [24] trained on ImageNet [6] of resolution of 256 × 256 . Following common practice [24, 30], we adopt a classifier-free guidance scale (CFG) of 1.5. According to [24], a widely used sampling method of the flow-based model is Heun's method [15]. In Table 1a, we demonstrate how our FlowTurbo can achieve faster inference than Heun's method in various computational budgets. Specifically, we conduct experiments with different sampling configurations (the second column of Table 1a), where we use the suffix to represent the number of Heun's method block ( H ), pseudo corrector block ( P ), and the velocity refiner block ( R ). Note that each Heun's block contains two evaluations of the velocity predictor while each pseudo corrector block only contains one. We also provide the total FLOPs during the sampling and the

inference speed of each sample configuration. In each group of comparison, we choose the sampling strategy of FlowTurbo to make the sampling quality (measured by the FID 50K ↓ ) similar to the baseline. Our results demonstrate that FlowTurbo can accelerate the inference by 37 . 2% ∼ 43 . 1% , while still achieving better sampling quality. Notably, FlowTurbo obtains 3.63 FID with a sampling speed of 41.6 ms/img, achieving real-time image generation.

Text-to-image generation. We adopt the 2-RF model in [20] as our base model for text-to-image generation. Note that we do not adopt the distilled version in [20] since we focus on accelerating flow-based models within the multi-step sampling paradigm. Following [20, 26], we compute the FID 5K ↓ between the generated 512 × 512 samples and the images on MS COCO 2017 [16] validation set. The results are summarized in Table 1b, where we compare the sampling speed/quality with the baseline Heun's method. Note that the notation of the sampling configuration is the same as Table 1a. The results clearly demonstrate that Our FlowTurbo can also achieve significant acceleration (29.8% ∼ 38.5%) on text-to-image generation.

4.3 Comparisons to State-of-the-Arts

In Table 2, we compare our FlowTurbo with state-of-theart methods on ImageNet 256 × 256 class-conditional generation. We use SiT-XL [24] as our base model and apply FlowTurbo with different sampling configurations on it. We show that FlowTurbo with H 1 P 5 R 3 achieves the sampling speed of 38 (ms / img) with 3.93 FID (still better than most methods like Mask-GIT [2], ADM [7]). On the other hand, FlowTurbo with H 8 P 9 R 5 archives the lowest FID 2.12, outperforming all the other methods. Besides, we also provide a comparison of the sampling speed/quality trade-offs of SiT (by changing the number of sampling steps of Heun's method) and FlowTurbo (by changing the sampling configurations) in 3, where the results of some other state-of-the-arts methods are also included. The comparison shows our FlowTurbo exhibits favorable sampling quality/speed trade-offs.

4.4 Analysis

Ablation of components of FlowTurbo. We evaluate the effectiveness of each component of FlowTurbo in Table 3a. Specifically, we start from the baseline, a 7-step Heun's method and gradually add components of FlowTurbo. In the sample config A, we show that adding a velocity refiner can significantly improve the FID ↓ ( 4 . 42 → 2 . 80 ), while introducing minimal computational costs (only +0 . 7% in the latency). From B to E, we adjust the ratios of Heun's method block, the pseudo corrector block, and the velocity refiner block to achieve different trade-offs between sampling speed and quality. In the last two rows, we show that our sample-aware compilation is better than standard model-level compilation, further increasing the sampling speed.

Choice of ∆ t . We find the choice of ∆ t during training is crucial and affects the sampling results a lot in our experiments, as shown in Table 3b. We find ∆ t ∈ (0 . 0 , 0 . 1] works well for more sampling steps like H 12 R 5 , while ∆ t ∈ [0 . 06 , 0 . 12] is better for fewer sampling steps like H 6 R 2 and H 9 R 3 . Besides, we find ∆ t ∈ (0 . 0 , 0 . 12] yields relatively good results in all the situations.

Effects of velocity refiner. We evaluate the effects of the different number of velocity refiners in Table 3c, and find that appropriately increasing the number of velocity refiners can improve the trade-off between sampling quality and speed. Specifically, we find H 6 R 2 can achieve better image quality and generation speed than the baseline H 8 .

Effects of pseudo corrector. In Table 3d, we fix the total number of both Heun's sample block and pseudo corrector block and adjust the ratio of the pseudo corrector. Our results demonstrate that increasing the number of pseudo corrector blocks can significantly improve the sampling speed while introducing neglectable performance drop ( e.g ., FlowTurbo with H 1 P 6 R 2 performs better than H 8 ).

40

100

1000

10000

Latency (ms / img)

1

2

3

4

5

6

7

8

9

FID 50K

Mask-GIT

ADM-G

LDM-4-G

DiT-XL

StyleGAN-XL

SiT

FlowTurbo

(c) Effects of the velocity refiner.

(a) Altering the order of H/P/R blocks.

Sample Config

FID

↓

H

5

P

7

R

3

P

5

H

P

5

7

R

H

7

5

R

3

H

R

7

P

3

3

2.24

2.38

13.66

30.94

Ablation of refiner architectures.

When designing the architecture for the velocity refiner, we followed a simple rule to make the refiner have a similar architecture as the base velocity predictor but with much fewer parameters ( 5% of the base model). The detailed architecture is described in Section 4.1 and Appendix C. For

(b) Repeating multiple

H/P/R

blocks

Latency

(ms / img)

72.4

79.2

53.7

60.4

example, since SiT consists of multiple transformer blocks, we simply use a single block as the refiner. For text-to-image generation, we reduce the number of layers and channels of the UNet. In our early experiments, we have tried another architecture for class-conditional image generation, where a SiT-S (a smaller version of the base velocity predictor SiT-XL) is adopted as the refiner (as shown in Table 5). We find that using a block of SiT-XL as the refiner is slightly better than the SiT-S. These results demonstrate that our framework is robust to the choice of model architectures for the velocity refiner.

Comparisons of different order of the blocks. According to the observation in Figure 1, the velocity during the sampling would become stable at the final few steps, where we adopted a lightweight refiner to regress the velocity offset. Besides, our pseudo corrector is designed to efficiently achieve

A moving train against the background of a blue sky and epic clouds

A pink rectangular potion with intricate gold adornment

An alien ship crashed with a sad alien sitting on the desert ground

A multicolored iridescent horse with unicorn horn and dragon wings

(a) Results of Heun's (2.6 s / img, left ) and FlowTurbo ( 1.8 s / img , right )

Image Inpainting

Image Editing

Object Removal

(b) Extensions

2-order convergence, which requires a 2-order intermediate result as initialization. This explains why we need several Heun's steps at the beginning. To further investigate how the order of the blocks would affect the sampling speed and quality, we perform experiments and summarize the results in Tables 4a and 4b. First, we find changing the order of the sampling blocks will cause worse sampling quality. Second, we show that sequentially using multiple blocks ( e.g ., [ H N H P N P R N R ] × k ) will cost the same inference time as H kN H P kN P R kN R but lead to worse visual quality.

Qualitative results and extensions. We provide qualitative results of high-resolution text-to-image generation by applying FlowTurbo to the newly released flow-based model Lumina-Next-T2I [9]. Since Lumina-Next-T2I adopts a heavy language model Gemma-2B [39] to extract text features and generates high-resolution images ( 1024 × 1024 ), the inference speed of it is slower than SiT [24]. In Figure 4a, we show that our FlowTurbo can generate images with better quality and higher inference speed compared with the baseline Heun's method. Besides, since FlowTurbo remains the multi-step sampling paradigm, it can be seamlessly applied to more applications like image inpainting, image editing, and object removal (Section 4.4). Please also refer to the Appendix C for the detailed implementation of various tasks.

Limitations and broader impact. Despite the effectiveness of FlowTurbo, our velocity refiner highly relies on the observation that the velocity is a 'stable value' during the sampling. However, we have not found such a stable value in diffusion-based models yet, which might limit the application. Besides, the abuse of FlowTurbo may also accelerate the generation of malicious content.

5 Conclusion

In this paper, we introduce FlowTurbo, a novel framework designed to accelerate flow-based generative models. By leveraging the stability of the velocity predictor's outputs, we propose a lightweight velocity refiner to adjust the velocity field offsets. This refiner comprises only about 5% of the original velocity predictor's parameters and can be efficiently trained in under 6 GPU hours. Additionally, we have proposed a pseudo corrector that reduces the number of model evaluations while maintaining the same convergence order as the second-order Heun's method. Furthermore, we propose a sample-aware compilation technique to enhance sampling speed. Extensive experiments on various flow-based generative models demonstrate FlowTurbo's effectiveness on both class-conditional image generation and text-to-image generation. We hope our work will inspire future efforts to accelerate flow-based generative models across various application scenarios.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62321005, Grant 624B1026, Grant 62336004, and Grant 62125603.

References

Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797 , 2023.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11315-11325, 2022.

Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692 , 2024.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 , 2023.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255. IEEE, 2009.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS , 34:8780-8794, 2021.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945 , 2024.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 33:68406851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS , 2021.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurIPS , 34:21696-21707, 2021.

Black Forest Labs. Flux: A powerful tool for text generation. https://huggingface.co/ black-forest-labs/FLUX.1-dev , 2024. Accessed: 2024-09-26.

John Denholm Lambert et al. Numerical methods for ordinary differential systems , volume 146. Wiley New York, 1991.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. Springer, 2014.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 , 2022.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. ICLR , 2022.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for highquality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740 , 2024.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14297-14306, 2023.

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2iadapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4296-4304, 2024.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR , 2022.

Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042 , 2023.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR , 2021.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 , 2024.

Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828 , 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , pages 3836-3847, 2023.

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. NeurIPS , 2023.

Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024.

A Detailed Background of Diffusion and Flow-based Models

In this section, we will provide a detailed background of diffusion and flow-based models, which is helpful to understand the difference and relationship between them.

A.1 Diffusion Models

The forward pass i.e. diffusioin pass of DPMs can be defined as a sequence of variables { x t } t ∈ [0 , 1] starting with x 0 , such that for any t ∈ [0 , 1] , x 0 ∈ R D is a D-dimensional random variable with an unknown data distribution p 0 ( x 0 ) . the distribution of x t conditioned on x 0 satisfies

where α t , σ t ∈ R + are differentiable functions of t with bounded derevatives. The choice for α t and σ t is referred to as the noise schedule of a DPM. Let p t ( x t ) denote the marginal distribution of x t , DPMs choose noise schedules to ensure the marginal distribution p 1 ( x 1 ) = N (0 , I ) and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictly decreasing w.r.t. t [13]. And we have

Moreover, Kingma et al . [13] prove that the following stochastic differential equation (SDE) has the same transition distribution q 0 t ( x t | x 0 ) as in (16) for any t ∈ [0 , 1] :

where w t ∈ R D is the standard Wiener process , and

Song et al. [38] have shown that the forward process in (18) has an equivalent reverse process from time 1 to 0 under some regularity conditions, starting with the marginal distribution p T ( x T ) :

where ¯ w t ∈ R D is a standard Wiener process in the reverse time. To solve the reverse process in (20), the only thing we should do is to estimate the score term ∇ x log p t ( x t ) at each time t. In practice, DPMs train a neural network ϵ θ ( x , t ) parameterized by θ to estimate the scaled score function: -σ t ∇ x log p t ( x t ) . The parameter θ is optimized by minimizing the following objective [10, 38, 24]

where λ ( t ) is a time-dependent coefficient. As ϵ θ ( x t , t ) can alse be regarded as predicting the Gaussian noise added to x t , it is usually called the noise prediction model . Since the ground truth of ϵ θ ( x t , t ) is -σ t ∇ x log p t ( x t ) , DPMs replace the score function in (20) by -ϵ θ ( x t , t ) /σ t and we refer to it as diffusion-based generative model. DPMs define a parameterized reverse process (diffusion SDE) from time 1 to 0 , starting with x 1 ∼ p 1 ( x 1 ) :

Samples can be generated from DPMs by solving the diffusion SDE in (22) with numerical solvers.

When discretizing SDEs, the step size is limited by the randomness of the Wiener process. A large step size (small number of steps) often causes non-convergence, especially in high dimensional spaces. For faster sampling, we can consider the associated probability flow ODE [38] which has the same marginla distribution at each time t as that of the SDE. Specifically, for DPMs, Song et al . [38] proved that the probability flow ODE of (22) is

Samples can be generated by solving the ODE from 1 to 0 . Comparing with SDEs, ODEs can be solved with larger step sizes as they have no randomness. Furthermore, we can take advantage of efficient numerical ODE solvers to accelerate the sampling.

A.2 Flow-based Models

To introduce flow in detail, first we construct a time-dependent vector field, u : [0 , 1] × R D → R D . A vector field u t can be used to construct a time-dependent diffeomorphic map, called a flow , ϕ : [0 , 1] × R D → R D ,defined via the ordinary differential equation (ODE):

Chen et al . [5] suggested modeling the vector field u t with a neural network v θ , which in turn leads to a deep parametric model of the flow ϕ t , called a Continuous Normalizing Flow (CNF). It is a more generic modeling technique and can capture the probability paths of diffusion process as well. Training a CNF becomes more practical since the propose of the conditional flow matching (CFM) technique [17], which learns the conditional velocity field of the flow.

For generative models, similar to (17) we can add some constraints to the noise schedule such that α 0 = 1 , σ 0 = 0 and α 1 = 0 , σ 1 = 1 , and then define the flow as:

The corresponding velocity vector field which is used to construct the flow ψ t can be represented as:

Consider the time-dependent probability density function (PDF) p t ( x ) of x t = ψ t ( x 0 | ϵ ) = α t x 0 + σ t ϵ . Lipman et al . [17] proved that the marginal vector field u t that generates the probability path p t satisfies a Partial Differential Equation (PDE) called continuity equation (also transport equation )

Using conditional flow matching technique v ( x t , t ) in (23) can be estimated parametrically as v θ ( x t , t ) by minimizing the following objective

We refer to (23) as a flow-based generative model. Since we have x t = ψ t ( x 0 | ϵ ) , the sampling of a flow-based model can be achieved by solving the probability flow ODE with learned velocity

A.3 Relationship Between Diffusion and Flow-based Models

There exists a straightforward connection between v ( x t , t ) and the score term σ t ∇ x log p t ( x t ) according to [24].

Algorithm 1 Heun's Method Sampler

Algorithm 2 Pseudo Corrector Sampler

We can define ζ t = ˙ σ t -˙ α t σ t α t , and we have ϵ θ ( x t , t ) to estimate -σ t ∇ x log p t ( x t ) , then derive the relationship between L DM ( θ ) and L FM ( θ ) We can plug (32) into the loss L FM ( θ ) in Equation (30)

Recall that

we can see that the difference of L DM ( θ ) and L FM ( θ ) during training is caused by the weighted function, which leaving to different trajectories and properties.

B Proof of Convergence of Pseudo Corrector

In this section, we will prove that the proposed pseudo corrector has the same local truncation error and global convergence order as Heun's method. The detailed sampling procedure of Heun's method

and pseudo corrector are provided in Algorithm 1 and Algorithm 2. In this section, we use x t i to represent the intermediate sampling result at the t i timestep, and use x ∗ t i = x ( t i ) to denote the corresponding ground-truth value on the trajectory. In all the proofs in this section, we omit the condition y for simplicity.

B.1 Assumptions

Assumption B.1. The velocity predictor v θ ( x , t ) is Lipschitz continous of constant L w.r.t x .

Assumption B.2. The velocity predictor v θ ( x , t ) has at least 2 derivatives d d t v θ ( x , t ) and d 2 d t 2 v θ ( x , t ) and the derivatives are continuous.

Assumption B.3. h = max 0 ≤ i ≤ N -1 h i = O (1 /N ) , where N is the total number of sampling steps.

All the above are common in the analysis of the convergence order of fast samplers [22, 23, 42] of diffusion models.

B.2 Local Convergence

We start by studying the local convergence and Heun's method. Considering the updating from t i to t i +1 and assume all previous results are correct (see the definition of local convergence [15]). The Taylor's expansion of x ∗ t i +1 at t i gives:

On the other hand, let ¯ x t i +1 be the prediction assuming x i is correct, the updating rule of Heun's method shows:

Therefore, the local truncation error can be computed by:

which indicates that Heun's method has 2 order of accuracy.

It is also noted that the local truncation error of the predictor step (which is the same as Euler's method) can be similarly derived by:

For pseudo corrector, the analysis of local convergence is the same since we need to assume all previous results (including the x t i and d i ), which means the local truncation error of pseudo corrector is the same as the Heun's method.

B.3 Global Convergence

Global convergence for Heun's method. When analyzing global convergence, we need to take into account both the local truncation error and the effects of the error of previous results. According to the Lipschitz condition, we have:

and

Combining the above two inequalities together, we have

where C 3 = hLC 2 2 + C 1 . Note that ∥ x ∗ t 0 -x t 0 ∥ = 0 (their is no error at the beginning of the sampling), it can be easily derived that

Therefore, we have proven that Heun's method have 2 order of global convergence.

Global convergence for pseudo corrector. The only difference between pseudo corrector and Heun's method is how d i is obtained. Pseudo corrector reuse the d i from the last sampling step rather than re-compute it as in Heun's method. As a result, d i used in pseudo corrector is computed on ˜ x t i rather than x t i , which will lead to another error term when analyzing the global convergence. Concretely, the global error of pseudo corrector can be computed by:

For the sake of simplicity, let ˜ ∆ i = ∥ x ∗ t i -˜ x t i ∥ and ∆ i = ∥ x ∗ t i -x t i ∥ . Therefore, the above formulas becomes:

By calculating (52) × hL + (53) we have:

Note that ∆ 0 + hL ˜ ∆ 0 = 0 . Let ∆ ′ i = ∆ i + hL ˜ ∆ i , we have

Similar to the derivation of (49), we can derive that

Therefore we have ∆ N ≤ C 9 h 2 , and thus the global convergence of pseudo corrector is 2-order.

which indicates that

C Implementation Details

Class-conditional image generation. We use the SiT-XL-2[24] as our base model to perform the experiments on class-conditional image generation. We use a single block of SiT-XL-2 as the Velocity Refiner. We double the input channel from 4 to 8 to take the previous velocity as input. The resulting velocity refiner only contains 29M parameters, about 4.3% of the original SiT-XL-2(675M). We use ImageNet-1K [6] 2 to train our velocity model. We used AdamW [21] optimizer for all models. We use a constant learning rate of 5 × 10 -5 and a batch size of 18 on a single A800 GPU. We used a random horizontal flip with a probability of 0.5 in data augmentation. We did not tune the learning rates, decay/warm-up schedules, AdamW parameters, or use any extra data augmentation during training. Our velocity refiner (for SiT-XL-2) trains at approximately 4.44 steps/sec on an A800 GPU, and converges in 30,000 steps, which takes about 2 hours.

Text-to-image generation. We use the 2-RF in InstaFlow [20] as our base model to perform the experiments on text-to-image generation. Since the architecture of the original velocity predictor in [20] is a U-Net [30], we cannot directly use a single block of it as the velocity refiner as we do for SiT [24]. Instead, we simply reduce the number of channels in each block from [320, 640, 1280, 1280] to [160, 160, 320, 320] and reduce the number of layers in each block from 2 to 1. We also double the input channel from 4 to 8 to take the previous velocity as input. The resulting velocity refiner only contains 43.5M parameters, about 5% of the original U-Net (860M). We use a subset of LAION [34] 3 containing only 50K images to train our velocity model. We use AdamW [21] optimizer with a learning rate of 2e-5 and weight decay of 0.0. We adopt a batch size of 16 and set the warming-up steps as 100. We also use a gradient clipping of 0.01 to stabilize training. We train our model on a single A800 GPU for 10K iterations, which takes about 5.5 hours.

Implementation of extension tasks. We have demonstrated our FlowTurbo is also suitable for extension tasks due to the multi-step nature of our framework in Section 4.4. For image inpainting, we adopt the inpainting pipeline in diffusion models 4 , where we merge the noise latent and the generated latent at a specific timestep by the input mask. For object removal, we first use a Grounded-SAM 5 to generate the mask and perform similar image inpainting pipeline. For image editing, we adopt the SDEdit [25] which first adds noise to the original image and use it as an intermediate result to continue the sampling.

D More Analysis

In this section, we provide more analysis through both quantitative results and qualitative results.

D.1 More Quantitative Results

Ablation of the number of the velocity refiners. In Table 6, we investigate how to choose the number of velocity refiners to get a better sampling quality. We adopt two basic configurations of H 7 R 10 and H 5 R 10 , and vary the number of velocity refiners from 3 to 6. We find that the FID will first decrease and then increase when N R becomes larger, and there exists an optimal N R = 5 where we reach the lowest FID. These results indicate that we can always tune this hyper-parameter to expect a better result.

More comparisons on text-to-image generation. In Table Table 7, we compare the sampling quality and speed of FLowTurbo with state-of-the-art diffusion models on text-to-image generation. For all the diffusion models, we adopt a 15-step DPM-Solver++ [23] as the default sampler. The FLOPs reported also take the multi-step sampling into account. Our results show that our FlowTurbo can achieve the lowest FID and inference latency.

D.2 More Qualitative Results

To better illustrate the sampling quality of our FlowTurbo, we provide more qualitative results on both class-conditional image generation and text-to-image generation.

Class-conditional image generation. We use SiT-XL [24] as our flow-based model for classconditional image generation. In Figure 5, we provide random samples from FlowTurbo of the sample config H 8 P 9 R 5 , which inference at 100 ms/img. We also demonstrate the sampling quality trade-offs in Figure 6, we compare the sampling quality of two different configurations H 1 P 5 R 3 (38 ms / img) and H 8 P 9 R 5 (100 ms / img). We generate the images from the same initial noise for better comparisons. Our result demonstrates that our FlowTurbo can achieve real-time image generation, and the sampling quality can be further improved with more computational budgets.

Text-to-image generation. We adopt Lumina-Next-T2I [9] to achieve text-to-image generation. We compare the sampling quality and speed of Heun's method and our FlowTurbo in Figure 7. We find that FlowTurbo can consistently generate images with better quality and more visual details, while requiring less inference time.

E Code

Our code is implemented in PyTorch 6 . We use the codebase of [24] to conduct experiments. The code is available at https://github.com/shiml20/FlowTurbo .

日.80日

00

istockphoio

VWACON

eEC

A beautiful panorama from inside a camper with beautiful beach and cliffs

The moon, a silver boat, sails through the sea of stars

A single brown bird perched on a mossy branch

Digital watercolor of a summer scape sunset, with flowery pastel colors

A legendary fruit castle in a fruit kingdom

Inside the glass sphere, Pirate Ship in a storm with waves in the dark

A beautiful girl flying through a paradox with piercing eyes

Underwater landscape with colorful mechanical parts, cable, wires

A cute kitten wearing a witch's robe and hat, holding a magic book

A tree house on a beautiful beach surrounded by mountains and waterfalls

A sailboat in the ocean with a moon in the sky

A rough linen knightess, wielding dual axes resembling two moons

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: the main contribution of this paper is a new framework to accelerate the sampling of flow-based generative models, which is fully discussed in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: the limitations are discussed in the last paragraph of Section 4.4

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: the theoretical result is the convergence of pseudo corrector, which is presented in Appendix B

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: we have included implementation details in both Section 4.1 and Appendix C. We have also included the code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: the datasets we used are all publicly available, we have included the code in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we have included these details in both Section 4.1 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: following common practice in [24, 30, 29, 4], we do not need to report error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: the compute resources have been discussed in Section 4.1 and Appendix C

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: we have carefully checked the NeurIPS Code of Ethics and make sure our work conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: the broader impacts are discussed in the last paragraph of Section 4.4

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: we propose an acceleration framework over existing flow-based generative models and thus we do not introduce new risks of misuse. The safeguards needs to be considered in the pre-trained flow-based generative models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: all the datasets are cited properly, and the licenses are also provided in Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: this paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: this paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FlowTurbo, a framework to accelerate flow-based generative models. The key contributions are:\n1) a lightweight velocity refiner to estimate the offset of velocity efficiently during sampling;\n2) a pseudo corrector to reduce the number of model evaluations while keeping the original second-order convergence;\n3) sample aware compilation to compile model evaluations, sampling steps and cfg into a static graph for extra speedup;\n\nExtensive experiments with both class-conditioned and text-to-image generation show significant acceleration (30-58%). Notably, on ImageNet 256x256 generations, FlowTurbo achieves the fastest sampling speed (38 ms/image) with FID 3.93. Moreover, due to the compatibility with multi-step sampling, it can also be applied to other tasks including image editing, inpainting, etc.\n\nSTRENGTHS:\nGiven the popularity of flow-based generation models, SD3 for image and Sora for video, flow-based model acceleration is an important area, which is under-explored compared with diffusion-based models. \n\nMotivated by the stability of velocity prediction during sampling, the authors proposed a lightweight velocity refiner. The pseudo corrector technique reduces the model evaluation while preserving the second-order convergency.  These ideas are novel and effective. The experimental results are comprehensive covering generation task with class-conditional and text-to-image generation, editing tasks including editing, inpainting, object removal. The ablation studies are thorough, examining each component's contribution and various hyperparameters. Finally, the code is also provided for reproducibility. The paper is easy to follow, for example, Figure 2 illustrates the motivation for light-weight estimation model very clearly.\n\nWEAKNESSES:\nFor text-to-image generation, insta-flow is adopted as the teacher model, which is built on top of sd1.5. It would be great to experiment with other flow-based models that are natively trained with flow matching, for example, sd3. \nIt would also be interesting to see how FlowTurbo performs when the model size changes. For example, SiT-XL vs SiT-L. \nFinally, one missing ablation is the architecture of refiner. how to design a refiner for different architectures?\n\nQUESTIONS:\nCan this method also be applied to video-based generation models? The inference is much slower for video generation compared with image generation. it would be great to see if there is a unified method for both image and video generation.\n\nLIMITATIONS:\nThe proposed method is limited to flow-based generation models.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores enhancing flow-based generative models for image generation by accelerating the sampling process while maintaining or improving image quality. The key contribution, FlowTurbo, is a new framework that introduces a lightweight velocity refiner to adjust velocity predictions during sampling, reducing computational costs. Additionally, techniques like pseudo correctors and sample-aware compilation further optimize the sampling speed. The results show significant acceleration in image generation tasks with improved FID scores, establishing a new benchmark for real-time image generation.\n\nSTRENGTHS:\nOverall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. This paper starts from the distinct features of flow models and naturally proposes corresponding solutions. Comprehensive ablations validate the effectiveness of the proposed methods.\n\nWEAKNESSES:\n- The design of three types of blocks in sampling is somewhat confusing to me. Why do you have to use these types of blocks and arrange them in this order? Do you have to empirically decide the order and numbers of these blocks for each task/domain/model?\n- The visualization in Figure 1 is interesting. However, visualizing the discretization error or curvature [1] during sampling seems to be a more suitable and accurate method.\n- Some important related works [1,2], tailored for flow models, should be discussed and compared.\n\n[1] Nguyen, Bao, Binh Nguyen, and Viet Anh Nguyen. \"Bellman optimal step-size straightening of flow-matching models.\" *arXiv preprint arXiv:2312.16414* (2023).\n\n[2] Shaul, Neta, et al. \"Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models.\" *arXiv preprint arXiv:2403.01329* (2024).\n\nQUESTIONS:\nIf I understand correctly, the proposed method can be applied to diffusion models as well. (Although Figure 1 illustrates flow models may be more suitable.) Have you tested the results on diffusion models like SD or PixArt?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose FlowTurbo, a method to adapt pre-trained flow-based generative models for faster sampling. The method is based on the observation that the predicted velocity field is quite stable throughout the integration time and hence at each timestamp only small refinement from the previous step is required. Therefore, the authors propose to train a lightweight refinement network and use it at certain timestamps instead of the original velocity field predictor. In addition to the refinement network the paper also suggests to speed up the Heun's integration method by reusing the velocities predicted at previous corrector steps. All the sampling blocks are compiled for further speed up. As a result, FlowTurbo demonstrates great trade-off between speed and quality compared to prior work and achieves sota results on certain tasks.\n\nSTRENGTHS:\nThe problem of speeding up generative models is of high importance nowadays. While the majority of the methods aims for decreasing the number of denoising steps, the authors of FlowTurbo propose to make each step significantly cheaper. This is achieved at low additional training costs and could have high impact on the methods that explicitly utilise the iterative nature of modern diffusion- or flow-based generative models (such as [1, 2, 3], see references in Weaknesses). The method is well-grounded and carefully ablated. The experiments demonstrate the effectiveness of FlowTurbo compared to default sampling.\n\nWEAKNESSES:\n1) **Prior work**: The paper does a good job with contextualizing it relative to the diffusion-based models. However, references to many flow-based methods are missing, which include:\n\n    - Models that leverage flow matching for generative process [2, 4, 5, 6, 7, 8, 9].\n\n    - And (which is more connected to the topic of the paper) the methods that try to speed up the sampling for flow-based models [10, 11, 12, 13].\n\n2) **Method**: Despite the good results, it seems that it is not trivial how to choose the exact configuration of the sampling blocks, i.e. the order and the number of different steps. From the first sight the configurations presented in the paper are quite random. At least some intuition on how to setup those is required. Otherwise, if the configuration needs to be tuned for every other dataset, the method is not that straight-forward to apply.\n\n3) **Evaluation**: \n    - The tables would be simpler to read if NFEs (number of function evaluations) were also included as a separate column.\n    - Is the step of the baseline method also compiled in tables 1a and 1b? If not, it would be nice to add this experiment, in addition to the ablations, to see the effect of other parts.\n\n[1] Watson et. al, Novel view synthesis with diffusion models, ICLR 2023\n\n[2] Davtyan et. al, Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV 2023\n\n[4] Hu et. al, Motion Flow Matching for Human Motion Synthesis and Editing, arxiv 2023\n\n[5] Fischer et. al, Boosting Latent Diffusion with Flow Matching, arxiv 2023\n\n[6] Davtyan et. al, Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation, AAAI 2024\n\n[7] Davtyan et. al, Enabling Visual Composition and Animation in Unsupervised Video Generation, arxiv 2024\n\n[8] Gui et. al, Depthfm: Fast monocular depth estimation with flow matching, arxiv 2024\n\n[9] Hu et. al, Flow Matching for Conditional Text Generation in a Few Sampling Steps, arxiv 2024\n\n[10] Shaul et. al, Bespoke Solvers for Generative Flow Models, arxiv 2023, ICLR 2024\n\n[11] Lee et. al, Minimizing Trajectory Curvature of ODE-based Generative Models, ICML 2023\n\n[12] Pooladian et. al, Multisample flow matching: Straightening flows with minibatch couplings, ICML 2023.\n\n[13] Tong et. al, Improving and generalizing flow-based generative models with minibatch optimal transport, TMLR 2023.\n\n[14] Lipman et. al, Flow matching for generative modeling, ICLR 2023.\n\nQUESTIONS:\n1) In the paper the order of blocks $H_{N_H}P_{N_P}R_{N_R}$ is fixed, and only the number of the blocks $N_H$, $N_P$ and $N_R$ changes. What is the reason behind this? Have you tried other schemes, e.g. sequentially using multiple blocks of type $H_{N_H}P_{N_P}R_{N_R}$?\n\n2) A rather nitpicky comment: Multiple times throughout the paper it is mentioned that the linear interpolation corresponds to the optimal transport from the noise distribution to the data distribution. It is not precisely correct, as the linear interpolation corresponds to the optimal transport from the noise distribution to the distribution concentrated around a particular data point [14]. Methods that attempt to steer flow-matching towards optimal transport between noise and data involve additional tricks [12, 13].\n\nSome typos:\n1) Line 137: \"interpolation between $x_0$ and $\\cancel{v}$ $\\epsilon$\".\n2) Line 170: \"simulate the $\\cancel{x_t}$  $x_{t_i}$\".\n3) Eq. 15: $\\cancel{x_i \\leftarrow x_{i-1}}$ $x_{t_i} \\leftarrow x_{t_{i - 1}}$\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new approach to accelerate the sampling process in flow-based generative models. Unlike diffusion models, flow-based models, which are based on learning velocity fields through flow-matching, have not seen extensive development in efficient sampling techniques. The authors introduce FlowTurbo, a framework that utilizes a lightweight velocity refiner to estimate velocity during sampling, significantly reducing computation time while maintaining or improving image quality. Additionally, the paper proposes pseudo corrector and sample-aware compilation techniques to further enhance sampling efficiency. Experimental results demonstrate that FlowTurbo achieves substantial speedups and high-quality results in both class-conditional and text-to-image generation tasks.\n\nSTRENGTHS:\n1. **Technical contribution**: The introduction of a lightweight velocity refiner to approximate the offset between the velocities between adjacent timesteps seems sound and is efficient as it requires only a light-weight model. \n   \n2. **Additional Techniques**: The integration of pseudo corrector and sample-aware compilation techniques shows a well-rounded approach to improving both speed and quality of the generative process.\n\n3. **Extensive Experiments**: The paper provides thorough experimental results, demonstrating the effectiveness of FlowTurbo across different tasks and models, including detailed comparisons with existing methods.\n\n4. **Real-time Performance**: Achieving real-time image generation with substantial improvements in inference time without compromising quality is a notable accomplishment.\n\n5. **Versatility**: FlowTurbo’s compatibility with various applications such as image editing and inpainting highlights its flexibility and potential for broader applications.\n\nWEAKNESSES:\n1. **Limited Comparison**: There are many efficient diffusion distillation methods, e.g., consistency distillation [1,2], consistent trajectory model [3], distribution matching distillation [4], adversarial approaches [5,6] (which also distilled using model trained with rectified flow). However, the paper has not been compared with any of those, which makes the comparison incomplete. In order to describe the technical advantage of using lightweight refiner, the author should compare with full fine-tuning (i.e., distillation) or with parameter efficient fine-tuning, e.g., LoRA. Thus, Table 2 should be filled with various baselines in order to precisely state ‘comparison with the state-of-the-arts’. \n2. **Complexity of implementation**: In addition, while the method seems sound, the method is somewhat complicated to use all Heun’s method, corrector step, etc. This might be an obstacle to scale this method to a large-scale.\n\n**References**\\\n[1] Song, Yang, et al. \"Consistency models.\" arXiv preprint arXiv:2303.01469 (2023). \\\n[2] Song, Yang, and Prafulla Dhariwal. \"Improved techniques for training consistency models.\" arXiv preprint arXiv:2310.14189 (2023).\\\n[3] Kim, Dongjun, et al. \"Consistency trajectory models: Learning probability flow ode trajectory of diffusion.\" arXiv preprint arXiv:2310.02279 (2023).\\\n[4] Yin, Tianwei, et al. \"One-step diffusion with distribution matching distillation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\\\n[5] Sauer, Axel, et al. \"Adversarial diffusion distillation.\" arXiv preprint arXiv:2311.17042 (2023).\\\n[6] Sauer, Axel, et al. \"Fast high-resolution image synthesis with latent adversarial diffusion distillation.\" arXiv preprint arXiv:2403.12015 (2024).\n\nQUESTIONS:\nMy major concern is that the paper considers improving the sampling efficiency of flow-based models which is good, but lacks comparison with other fast-sampling diffusion models. Given the theoretical or practical similarities between diffusion models and flow matching models, I believe the previous methods can be also applied to flow matching models. In that sense, there is a large gap in terms of performance, as previous works can sample within only 2-4 steps, e.g., consistency models. In order to elaborate the author’s claim that the velocity refiner is efficient and as good as those methods, the author should provide some additional comparison with those methods.\n\nLIMITATIONS:\nThe authors mentioned some limitations that those method could not be used for diffusion models, yet it could be in a better version if they demonstrate with detailed analysis.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}