FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner

Authors: wenliang  zhao
minglei  shi
xumin  yu
jie  zhou
jiwen  lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 05:08:36

Abstract: Building on the success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc. By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1\%$\sim$58.3\% on class-conditional generation and 29.8\%$\sim$38.5\% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-art. Code is available at https://github.com/shiml20/FlowTurbo.

Main paper: 
 introduction: 
 In recent years, diffusion models have emerged as powerful generative models, drawing considerable interest and demonstrating remarkable performance across various domains [10, 38, 30, 12] . Diffusion models utilize a denoising network, ϵ θ , to learn the reverse of a diffusion process that gradually adds noise to transform the data distribution into a Gaussian distribution. While the formulation of diffusion models enables stable training and flexible condition injection [30] , sampling from these models requires iterative denoising. This process necessitates multiple evaluations of the denoising network, thereby increasing computational costs. To address this, several techniques such as fast diffusion samplers [22, 18, 42] and efficient distillation [31, 37] have been proposed to reduce the sampling steps of diffusion models.Alongside the research on diffusion models, flow-based models [5, 19, 17] have garnered increasing attention due to their versatility in modeling data distributions. Flow is defined as a probability path that connects two distributions and can be efficiently modeled by learning a neural network to estimate the conditional velocity field through a neural network v θ via flow matching [17] . Encompassing the standard diffusion process as a special case, flow-based generative models support more flexible choices of probability paths. Recent work has favored a simple linear interpolant path [20, 24, 8] , which corresponds to the optimal transport from the Gaussian distribution to the data distribution. This linear connection between data and noise results in a more efficient sampling process for flowbased models. However, unlike diffusion models, which benefit from numerous efficient sampling methods, current samplers for flow-based models primarily rely on traditional numerical methods such as Euler's method and Heun's method [24] . These traditional methods, while functional, fail to fully exploit the unique properties of flow-based generative models, thereby limiting the potential for faster and more efficient sampling.In this paper, we propose FlowTurbo, a framework designed to accelerate the generation process of flow-based generative models. FlowTurbo is motivated by comparing the training objectives of diffusion and flow-based generative models, as well as analyzing how the prediction results ϵ θ and v θ vary over time. Our observation, illustrated in Figure 1 , indicates that the velocity predictions of a flow-based model remain relatively stable during sampling, in contrast to the more variable predictions of ϵ θ in diffusion models. This stability allows us to regress the offset of the velocity at each sampling step using a lightweight velocity refiner, which contains only 5% of the parameters of the original velocity prediction model. During the sampling process, we can replace the original velocity prediction model with our lightweight refiner at specific steps to reduce computational costs.As a step towards real-time image generation, we propose two useful techniques called pseudo corrector and sample-aware compilation to further improve the sampling speed. Specifically, the pseudo corrector method modifies the updating rule in Heun's method by reusing the velocity prediction of the previous sampling step, which will reduce the number of model evaluations at each step by half while keeping the original convergence order. The sample-aware compilation integrates the model evaluations, the sampling steps as well as the classifier-free guidance [11] together and compile them into a static graph, which can bring extra speedup compared with standard model-level compilation. Since each sample block is independent, we can still adjust the number of inference steps and sampling configurations flexibly.Our FlowTurbo framework is fundamentally different from previous one-step distillation methods for diffusion models [20, 40, 32] , which require generating millions of noise-image pairs offline and conducting distillation over hundreds of GPU days. In contrast, FlowTurbo's velocity refiner can be efficiently trained on pure images in less than 6 hours. Moreover, one-step distillation-based methods are limited to image generation and disable most of the functionalities of the original base model. Conversely, FlowTurbo preserves the multi-step sampling paradigm, allowing it to be effectively applied to various tasks such as image editing, inpainting, and more.We perform extensive experiments to evaluate our method. By applying FlowTurbo to different flow-based models, we obtain an acceleration ratio of 53.1%∼58.3% on class-conditional generation and 29.8%∼38.5% on text-to-image generation. Notably, FlowTurbo attains an FID score of 2.12 on ImageNet with 100 (ms / img) and FID of score 3.93 with 38 (ms / img), thereby enabling realtime image generation and establishes the new state-of-the-art. Additionally, we present qualitative comparisons demonstrating how FlowTurbo generates superior images with higher throughput and how it can be seamlessly integrated into various applications such as image editing, inpainting, etc. We believe our FlowTurbo can serve as a general framework to accelerate flow-based generative models and will see wider use as these models continue to grow [24, 20, 8, 9] .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions scope?Answer: [Yes] Justification: the main contribution of this paper is a new framework to accelerate the sampling of flow-based generative models, which is fully discussed in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Diffusion and flow-based models. Diffusion models [10, 38] are a family of generative models that have become the de-facto method for high-quality generation. The diffusion process gradually adds noise to transform the data distribution to a normal distribution, and the goal of diffusion models is to use a network ϵ θ to learn the reverse of the diffusion process via score-matching [10, 38] . Rombach et Figure 1 : Visualization of the curvatures of the sampling trajectories of different models. We compare the curvatures of the model predictions of a standard diffusion model (DiT [28] ) and several flow-based models (SiT [24] , SD3-Medium [8] , FLUX.1-dev [14] , and Open-Sora [43] ) during the sampling. We observe that the v θ in flow-based models is much more stable than ϵ of diffusion models during the sampling, which motivates us to seek a more lightweight estimation model to reduce the sampling costs of flow-based generative models.al. [30] first scales up diffusion models to large-scale text-to-image generation by performing the diffusion on latent space and adopting cross-attention to inject conditions. The pre-trained diffusion models can also be easily fine-tuned to achieve generation with more diverse conditions [41, 27] and have attracted increasing attention in the community. Flow-based generative models are different from diffusion models in both data modeling and training objectives. Flow-based models [20, 17, 8, 24] consider the probability path from one distribution to another, and learn the velocity field via flow matching [17] . By choosing the linear interpolant as the probability path which corresponds to the optimal transport from the normal distribution to the data distribution, the trajectory from noise to data becomes more straighter which is beneficial to the sampling. Recent work [24, 8] have demonstrates the effectiveness and scalability of flow-based generation models. However, both diffusion and flow-based models requires multiple evaluations of the prediction model, leading to lower inference speed than traditional architectures like GAN. In this work, we focus on this issue and aim to accelerate flow-based generative models.Efficient visual generation. Accelerating the generation of diffusion models has become an increasingly important topic. Existing methods can be roughly categorized as training-free and training-based methods. Training-free methods aim to design faster samplers that can reduce the approximation error when sampling from the diffusion SDE or ODE [36, 22, 18, 42] , while keeping the weights of diffusion models unchanged. Training-based methods often aim to reshape the sampling trajectory by distillation from the diffusion model [31, 40] to achieve the few-step or even one-step generation. These training-based methods usually requires multiple-round of distillation [31, 20] and expensive training resources (e.g., >100 GPU days in [20] ). Besides, the distilled one-step model no longer supports image editing due to the lack of multi-step sampling. Although there are a variety of methods for accelerating diffusion models, there are few fast sampling methods designed for flow-based generative models. Existing flow-based models adopt traditional numerical methods like Euler's method or Heun's method during the inference [24] . In this work, we provide a framework called FlowTurbo to accelerate the generation of flow-based models by learning a lightweight velocity refiner (which only requires <6 GPU hours) to regress the offset of the velocity. Together with other proposed techniques, FlowTurbo addresses the previously unmet need for an efficient flow-based generation framework, paving the way for real-time generative applications.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: the limitations are discussed in the last paragraph of Section 4.4Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: diffusion and flow-based models: 
 Diffusion models. Recently, diffusion models [10, 38, 35, 30] have emerged as a powerful family of generative models. The diffusion models are trained to learn the inverse of a diffusion process such that it can recover the data distribution p 0 (x 0 ) from the Gaussian noise. The diffusion process can be represented as:EQUATIONwhere α t , σ t are the chosen noise schedule such that the marginal distribution p 1 (x 1 ) ∼ N (0, I).The optimization of diffusion models can be derived by either minimizing the ELBO of the reverse process [10] or solving the reverse diffusion SDE [38] , which would both lead to the same training objective of score-matching, i.e., to learn a noise prediction model ϵ θ (x t , t) to estimate the scaled score function -σ t ∇ x log p t (x t ):EQUATIONwhere λ(t) is a time-dependent coefficient. Sampling from a diffusion model can be achieved by solving the reverse-time SDE or the corresponding diffusion ODEs [38] , which can be efficiently achieved by modern fast diffusion samplers [36, 22, 42] .Flow-based models. Flow-based models can be traced back to Continuous Normalizing Flows [5] (CNF), which is a more generic modeling technique and can capture the probability paths of the diffusion process as well [17] . Training a CNF becomes more practical since the purpose of the flow matching technique [17] , which learns the conditional velocity field of the flow. Similar to (1), we can add some constraints to the noise schedule such that α 0 = 1, σ 0 = 0 and α 1 = 0, σ 1 = 1, and then define the flow as:EQUATIONIn this case, the velocity field that generates the flow ψ t can be represented as:EQUATIONThe training objective of conditional flow matching is to train a velocity prediction model v θ to estimate the conditional velocity field:EQUATIONThe sampling of a flow-based model can be achieved by solving the probability flow ODE with the learned velocityEQUATIONSince the formulation of the flow ψ t can be viewed as the interpolation between x 0 and v, it is also referred to as interpolant in some literature [1, 24] . Among various types of interpolants, a very simple choice is linear interpolant [24, 8] , where α t = (1 -t) and σ t = t. In this case, the velocity field becomes a straight line connecting the initial noise and the data point, which also corresponds to the optimal transport between the two distributions [19, 17] . The effectiveness and scalability of the linear interpolant have also been proven in recent work [17, 24, 8] .We consider the velocity estimation in flow-based generative models with the linear interpolant [24, 8, 20] . As shown in (5) , the training target of the velocity prediction model v θ is exactly ϵx 0 , a constant value independent of t. Our main motivation is to efficiently estimate the velocity during the sampling, instead of evaluating the whole velocity prediction model v θ every time.Analyzing the stability of velocity. We start by analyzing the stability of the output value of v θ along the sampling trajectory. By comparing the training objectives of diffusion and flow-based models (2)(5), we know that the target of v θ is independent of t. A more in-depth discussion is provided in Appendix A.3, where we show the two training objectives have different time-dependent weight functions. To verify whether there are similar patterns during the sampling, we compare how the prediction results change across the sampling steps in Figure 1 . Specifically, we compare the curvatures of ϵ θ of a diffusion model (DiT [28] ) and the v θ of flow-based models (SiT [8] , SD3 [8] , etc) during the sampling steps. than the ϵ θ of a diffusion model. Therefore, We define the v θ as a "stable value". The stability of v θ makes it possible to obtain the velocity more efficiently rather than performing the forward pass of the whole velocity prediction network v θ at every sampling step.Learning a lightweight velocity refiner. Since the velocity in a flow-based model is a "stable value", we propose to learn a lightweight refiner that can adjust the velocity with minimal computational costs. The velocity refiner takes as inputs both the current intermediate result and the velocity of the previous step, and returns the offset of velocity:EQUATIONThe velocity refiner r ϕ can be designed to be very lightweight (< 5% parameters of v θ ). The detailed architecture can be found in Appendix C.To learn the velocity refiner, we need to minimize the difference between the output of r ϕ and the actual offset v ti -v ti-1 . However, it requires multiple-step sampling to obtain an intermediate result x ti to make the training objective perfectly align with our target. To reduce the training cost, we simulate the x t with one-step sampling starting from x ti-1 , which is directly obtained by the flow ψ ti-1 . The detailed procedure to compute the loss is listed as follows:EQUATIONEQUATION)Where ∆t = t i -t i-1 and we use a simple Euler step for the Solver to obtain x ti . Once the velocity refiner is learned, we can use it to replace the original v θ at some specific sampling steps. We will demonstrate through experiments that adding the velocity refiner can improve the sampling quality without introducing noticeable computational overhead.Compatibility with classifier-free guidance. Classifier-free guidance [11] is a useful technique to improve the sampling quality in conditional sampling. Let y be the condition, the classifier-free guidance for a velocity prediction model [8] can be defined as:EQUATION)where ζ is the guidance scale and ∅ denotes the null condition. To make our velocity refiner support classifier-free guidance, we only need to make sure both the conditional prediction v θ (x, t|y) and the unconditional prediction v θ (x, t|∅) appear during the training. Note that we always feed the velocity prediction model v θ and the velocity refiner r ϕ with the same condition.EQUATIONEQUATIONwhere we set the γ 1 = 0.1 as the probability of using an unconditional velocity.The sampling costs of a flow-based model can be significantly minimized by integrating our lightweight velocity refiner r ϕ in place of the velocity prediction network v θ at selected sampling steps. In this section, we propose two techniques to further improve the sampling speed towards real-time image generation. Pseudo corrector. Traditional numerical ODE solvers are usually used to sample from a probability flow ODE. For example, SiT [8] adopt a Heun method (or improved Euler's method) [15] as the ODE solver. The update rule from t i-1 to t i can be written as:EQUATIONEQUATIONEach Heun step contains a predictor step ( 14) and a corrector step (15) , thus includes two evaluations of the velocity predictor v θ , bringing extra inference costs. Motivated by [42] , we propose to reuse the d i in the next sampling step, instead of re-computing it viad i ← v θ (x ti , t i |y) (see Figure 2 (b)(c) for illustration).We call this a pseudo corrector since it is different from the predictor-corrector solvers in numerical analysis. It can be proved (see Appendix B) that the pseudo corrector also enjoys 2-order convergence while only having one model evaluation at each step.Sample-aware compilation. Compiling the network into a static graph is a widely used technique for model acceleration. However, all the previous work only considers network-level compilation, i.e., only compiling the ϵ θ or v θ . We propose the sample-aware compilation which wraps both the forward pass of v θ or r ϕ and the sampling operation together (including the classifier-free guidance) and performs the compilation. For example, the sample blocks illustrated in Figure 2 (b, c ) are compiled into static graphs. Since each sample block is independent, we can still adjust the number of inference steps and sampling configurations flexibly.Recently, there have been more and more training-based methods [20, 40, 32] aiming to accelerate diffusion models or flow-based models through one-step distillation. Although these methods can achieve faster inference, they usually require generating paired data using the pre-trained model and suffer from large training costs (e.g., >100 GPU days in [40, 20] ). Besides, one-step methods only keep the generation ability of the original model while disabling more diverse applications such as image inpainting and image editing. In contrast, our FlowTurbo aims to accelerate flow-based models through velocity refinement, which still works in a multi-step manner and performs sampling on the original trajectory. For example, FlowTurbo can be easily combined with existing diffusion-based image editing methods like SDEdit [25] (see Section 4.4).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 We conduct extensive experiments to verify the effectiveness of FlowTurbo. Specifically, we apply FlowTurbo to both class-conditional image generation and text-to-image generation tasks and demonstrate that FlowTurbo can significantly reduce the sampling costs of the flow-based generative models. We also provide a detailed analysis of each component of FlowTurbo, as well as qualitative comparisons of different tasks.In our experiments, we consider two widely used benchmarks including class-conditional image generation and text-to-image generation. For class-conditional image generation, we adopt a transformer-Table 1 : Main results. We apply our FlowTurbo on SiT-XL [24] and the 2-RF of InstaFlow [20] to perform class-conditional image generation and text-to-image generation, respectively. The image quality is measured by the FID 50K↓ on ImageNet (256×256) and the FID 5K↓ on MS COCO 2017 (512×512). We use the suffix to represent the number of Heun's method block (H), pseudo corrector block (P ), and the velocity refiner block (R). Our results demonstrate that FlowTurbo can significantly accelerate the inference of flow-based models while achieving better sampling quality.(a) Class-conditional Image Generation style flow-based model SiT-XL [24] pre-trained on ImageNet 256×256. For text-to-image generation, we utilize InstaFlow [20] as the flow-based model, whose backbone is a U-Net similar to Stable-Diffusion [30] . Note that we use the 2-RF model from [19] instead of the distilled version since our FlowTurbo is designed to achieve acceleration within the multi-step sampling framework. The velocity refiner only contains 4.3% and 5% parameters of the corresponding predictor, and the detailed architecture can be found in Appendix C. During training, we randomly sample ∆t ∈ (0, 0.12] and compute the training objectives in (13) . In both tasks, we use a single NVIDIA A800 GPU to train the velocity refiner and find it converges within 6 hours. We use a batch size of 8 on a single A800 GPU to measure the latency of each method. Please refer to Appendix C for more details.Class-conditional image generation. We adopt the SiT-XL [24] trained on ImageNet [6] of resolution of 256 × 256. Following common practice [24, 30] , we adopt a classifier-free guidance scale (CFG) of 1.5. According to [24] , a widely used sampling method of the flow-based model is Heun's method [15] . In Table 1a , we demonstrate how our FlowTurbo can achieve faster inference than Heun's method in various computational budgets. Specifically, we conduct experiments with different sampling configurations (the second column of Table 1a ), where we use the suffix to represent the number of Heun's method block (H), pseudo corrector block (P ), and the velocity refiner block (R). Note that each Heun's block contains two evaluations of the velocity predictor while each pseudo corrector block only contains one. We also provide the total FLOPs during the sampling and the inference speed of each sample configuration. In each group of comparison, we choose the sampling strategy of FlowTurbo to make the sampling quality (measured by the FID 50K↓) similar to the baseline. Our results demonstrate that FlowTurbo can accelerate the inference by 37.2% ∼ 43.1%, while still achieving better sampling quality. Notably, FlowTurbo obtains 3.63 FID with a sampling speed of 41.6 ms/img, achieving real-time image generation.Text-to-image generation. We adopt the 2-RF model in [20] as our base model for text-to-image generation. Note that we do not adopt the distilled version in [20] since we focus on accelerating flow-based models within the multi-step sampling paradigm. Following [20, 26] , we compute the FID 5K↓ between the generated 512 × 512 samples and the images on MS COCO 2017 [16] validation set. The results are summarized in Table 1b , where we compare the sampling speed/quality with the baseline Heun's method. Note that the notation of the sampling configuration is the same as Table 1a . The results clearly demonstrate that Our FlowTurbo can also achieve significant acceleration (29.8%∼38.5%) on text-to-image generation. In Table 2 , we compare our FlowTurbo with state-of-theart methods on ImageNet 256 × 256 class-conditional generation. We use SiT-XL [24] as our base model and apply FlowTurbo with different sampling configurations on it. We show that FlowTurbo with H 1 P 5 R 3 achieves the sampling speed of 38 (ms / img) with 3.93 FID (still better than most methods like Mask-GIT [2] , ADM [7] ). On the other hand, FlowTurbo with H 8 P 9 R 5 archives the lowest FID 2.12, outperforming all the other methods. Besides, we also provide a comparison of the sampling speed/quality trade-offs of SiT (by changing the number of sampling steps of Heun's method) and FlowTurbo (by changing the sampling configurations) in 3, where the results of some other state-of-the-arts methods are also included. The comparison shows our FlowTurbo exhibits favorable sampling quality/speed trade-offs.Ablation of components of FlowTurbo. We evaluate the effectiveness of each component of FlowTurbo in Table 3a . Specifically, we start from the baseline, a 7-step Heun's method and gradually add components of FlowTurbo. In the sample config A, we show that adding a velocity refiner can significantly improve the FID↓ (4.42 → 2.80), while introducing minimal computational costs (only +0.7% in the latency). From B to E, we adjust the ratios of Heun's method block, the pseudo corrector block, and the velocity refiner block to achieve different trade-offs between sampling speed and quality. In the last two rows, we show that our sample-aware compilation is better than standard model-level compilation, further increasing the sampling speed.Choice of ∆t. We find the choice of ∆t during training is crucial and affects the sampling results a lot in our experiments, as shown in Table 3b . We find ∆t ∈ (0.0, 0.1] works well for more sampling steps like H 12 R 5 , while ∆t ∈ [0.06, 0.12] is better for fewer sampling steps like H 6 R 2 and H 9 R 3 . Besides, we find ∆t ∈ (0.0, 0.12] yields relatively good results in all the situations. Effects of velocity refiner. We evaluate the effects of the different number of velocity refiners in Table 3c , and find that appropriately increasing the number of velocity refiners can improve the trade-off between sampling quality and speed. Specifically, we find H 6 R 2 can achieve better image quality and generation speed than the baseline H 8 .Effects of pseudo corrector. In Table 3d , we fix the total number of both Heun's sample block and pseudo corrector block and adjust the ratio of the pseudo corrector. Our results demonstrate that increasing the number of pseudo corrector blocks can significantly improve the sampling speed while introducing neglectable performance drop (e.g., FlowTurbo with H 1 P 6 R 2 performs better than H 8 ). Table 3 : Ablation studies. We evaluate the effectiveness of each component in FlowTurbo as well as the selection of some hyper-parameters. (a) We gradually add the components of FlowTurbo to the baseline and show that FlowTurbo can achieve over 50% acceleration with better sampling quality. (b) we experiment with different ranges of ∆t and find ∆t ∈ (0.0, 0.12] yields relatively good results in all the situations. (c)(d) we show how the sampling quality/speed changes with the number of velocity refiner and pseudo corrector blocks.(a) Ablation of components of FlowTurbo. Ablation of refiner architectures.When designing the architecture for the velocity refiner, we followed a simple rule to make the refiner have a similar architecture as the base velocity predictor but with much fewer parameters ( 5% of the base model). The detailed architecture is described in Section 4.1 and Appendix C. For example, since SiT consists of multiple transformer blocks, we simply use a single block as the refiner. For text-to-image generation, we reduce the number of layers and channels of the UNet. In our early experiments, we have tried another architecture for class-conditional image generation, where a SiT-S (a smaller version of the base velocity predictor SiT-XL) is adopted as the refiner (as shown in Table 5 ). We find that using a block of SiT-XL as the refiner is slightly better than the SiT-S. These results demonstrate that our framework is robust to the choice of model architectures for the velocity refiner.Comparisons of different order of the blocks. According to the observation in Figure 1 , the velocity during the sampling would become stable at the final few steps, where we adopted a lightweight refiner to regress the velocity offset. Besides, our pseudo corrector is designed to efficiently achieve With better image quality, our method requires much less sampling time (-30.8%). (b) Since FlowTurbo remains the multi-step sampling paradigm, it can be seamlessly applied to more applications such as image inpainting, image editing, and object removal.2-order convergence, which requires a 2-order intermediate result as initialization. This explains why we need several Heun's steps at the beginning. To further investigate how the order of the blocks would affect the sampling speed and quality, we perform experiments and summarize the results in Tables 4a and 4b . First, we find changing the order of the sampling blocks will cause worse sampling quality. Second, we show that sequentially using multiple blocks (e.g., [H N H P N P R N R ] ×k ) will cost the same inference time as H kN H P kN P R kN R but lead to worse visual quality.Qualitative results and extensions. We provide qualitative results of high-resolution text-to-image generation by applying FlowTurbo to the newly released flow-based model Lumina-Next-T2I [9] . Since Lumina-Next-T2I adopts a heavy language model Gemma-2B [39] to extract text features and generates high-resolution images (1024 × 1024), the inference speed of it is slower than SiT [24] . In Figure 4a , we show that our FlowTurbo can generate images with better quality and higher inference speed compared with the baseline Heun's method. Besides, since FlowTurbo remains the multi-step sampling paradigm, it can be seamlessly applied to more applications like image inpainting, image editing, and object removal (Section 4.4). Please also refer to the Appendix C for the detailed implementation of various tasks.Limitations and broader impact. Despite the effectiveness of FlowTurbo, our velocity refiner highly relies on the observation that the velocity is a "stable value" during the sampling. However, we have not found such a stable value in diffusion-based models yet, which might limit the application. Besides, the abuse of FlowTurbo may also accelerate the generation of malicious content. 
conclusion: 
 In this paper, we introduce FlowTurbo, a novel framework designed to accelerate flow-based generative models. By leveraging the stability of the velocity predictor's outputs, we propose a lightweight velocity refiner to adjust the velocity field offsets. This refiner comprises only about 5% of the original velocity predictor's parameters and can be efficiently trained in under 6 GPU hours. Additionally, we have proposed a pseudo corrector that reduces the number of model evaluations while maintaining the same convergence order as the second-order Heun's method. Furthermore, we propose a sample-aware compilation technique to enhance sampling speed. Extensive experiments on various flow-based generative models demonstrate FlowTurbo's effectiveness on both class-conditional image generation and text-to-image generation. We hope our work will inspire future efforts to accelerate flow-based generative models across various application scenarios. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: we have carefully checked the NeurIPS Code of Ethics and make sure our work conforms with it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: the broader impacts are discussed in the last paragraph of Section 4.4Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: we propose an acceleration framework over existing flow-based generative models and thus we do not introduce new risks of misuse. The safeguards needs to be considered in the pre-trained flow-based generative models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: all the datasets are cited properly, and the licenses are also provided in Appendix C.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we will provide a detailed background of diffusion and flow-based models, which is helpful to understand the difference and relationship between them.The forward pass i.e. diffusioin pass of DPMs can be defined as a sequence of variables {x t } t∈[0,1] starting with x 0 , such that for any t ∈ [0, 1], x 0 ∈ R D is a D-dimensional random variable with an unknown data distribution p 0 (x 0 ). the distribution of x t conditioned on x 0 satisfiesEQUATIONwhere α t , σ t ∈ R + are differentiable functions of t with bounded derevatives. The choice for α t and σ t is referred to as the noise schedule of a DPM. Let p t (x t ) denote the marginal distribution of x t , DPMs choose noise schedules to ensure the marginal distribution p 1 (x 1 ) = N (0, I) and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictly decreasing w.r.t. t [13] . And we haveEQUATIONMoreover, Kingma et al. [13] prove that the following stochastic differential equation (SDE) has the same transition distribution q 0t (x t |x 0 ) as in ( 16) for any t ∈ [0, 1]:EQUATIONwhere w t ∈ R D is the standard Wiener process , andEQUATIONSong et al. [38] have shown that the forward process in ( 18) has an equivalent reverse process from time 1 to 0 under some regularity conditions, starting with the marginal distribution p T (x T ):EQUATIONwhere wt ∈ R D is a standard Wiener process in the reverse time. To solve the reverse process in (20) , the only thing we should do is to estimate the score term ∇ x logp t (x t ) at each time t. In practice, DPMs train a neural network ϵ θ (x, t) parameterized by θ to estimate the scaled score function: -σ t ∇ x log p t (x t ). The parameter θ is optimized by minimizing the following objective [10, 38, 24 ]EQUATIONwhere λ(t) is a time-dependent coefficient. As ϵ θ (x t , t) can alse be regarded as predicting the Gaussian noise added to x t , it is usually called the noise prediction model . Since the ground truth of ϵ θ (x t , t) is -σ t ∇ x logp t (x t ), DPMs replace the score function in (20) by -ϵ θ (x t , t)/σ t and we refer to it as diffusion-based generative model. DPMs define a parameterized reverse process (diffusion SDE) from time 1 to 0, starting with x 1 ∼ p 1 (x 1 ):EQUATIONSamples can be generated from DPMs by solving the diffusion SDE in (22) with numerical solvers.When discretizing SDEs, the step size is limited by the randomness of the Wiener process. A large step size (small number of steps) often causes non-convergence, especially in high dimensional spaces. For faster sampling, we can consider the associated probability flow ODE [38] which has the same marginla distribution at each time t as that of the SDE. Specifically, for DPMs, Song et al. [38] proved that the probability flow ODE of ( 22) isEQUATIONSamples can be generated by solving the ODE from 1 to 0. Comparing with SDEs, ODEs can be solved with larger step sizes as they have no randomness. Furthermore, we can take advantage of efficient numerical ODE solvers to accelerate the sampling.To introduce flow in detail, first we construct a time-dependent vector field, u :[0, 1] × R D → R D .A vector field u t can be used to construct a time-dependent diffeomorphic map, called a flow, ϕ : [0, 1] × R D → R D ,defined via the ordinary differential equation (ODE):EQUATIONEQUATIONChen et al. [5] suggested modeling the vector field u t with a neural network v θ , which in turn leads to a deep parametric model of the flow ϕ t , called a Continuous Normalizing Flow (CNF). It is a more generic modeling technique and can capture the probability paths of diffusion process as well.Training a CNF becomes more practical since the propose of the conditional flow matching (CFM) technique [17] , which learns the conditional velocity field of the flow.For generative models, similar to (17) we can add some constraints to the noise schedule such that α 0 = 1, σ 0 = 0 and α 1 = 0, σ 1 = 1, and then define the flow as:EQUATIONThe corresponding velocity vector field which is used to construct the flow ψ t can be represented as:EQUATIONConsider the time-dependent probability density function (PDF) p t (x) of x t = ψ t (x 0 |ϵ) = α t x 0 + σ t ϵ. Lipman et al. [17] proved that the marginal vector field u t that generates the probability path p t satisfies a Partial Differential Equation (PDE) called continuity equation (also transport equation)EQUATIONUsing conditional flow matching technique v(x t , t) in ( 23) can be estimated parametrically as v θ (x t , t) by minimizing the following objectiveEQUATIONWe refer to ( 23) as a flow-based generative model. Since we have x t = ψ t (x 0 |ϵ), the sampling of a flow-based model can be achieved by solving the probability flow ODE with learned velocityEQUATIONThere exists a straightforward connection between v(x t , t) and the score term σ t ∇ x log p t (x t ) according to [24] .EQUATIONAlgorithm 1 Heun's Method SamplerRequire: timesteps {t i } N -1 i=0 , α t , σ t , x 0 ∼ N (0, I), velocity prediction model v θ (x, t|y)for i = 0 to N -1 do ∆t i ← t i+1 -t i d i ← v θ (x i , t i |y) xti+1 ← x i + ∆t i d i d i+1 ← v θ (x ti+1 , t i+1 |y) x ti+1 ← x i + ∆ti 2 [d i + d i+1 ] end for return: x N Algorithm 2 Pseudo Corrector Sampler Require: timesteps {t i } N -1 i=0 , α t , σ t , x 0 ∼ N (0, I), velocity prediction model v θ (x, t|y) ∆t ← t 1 -t 0 for i = 0 to N -1 do ∆t i ← t i+1 -t i if i = 0 then d i ← v θ (x i , t i |y) end if xti+1 ← x i + ∆t i d i d i+1 ← v θ (x ti+1 , t i+1 |y) x ti+1 ← x i + ∆ti 2 [d i + d i+1 ] end for return: x N We can define ζ t = σt -αtσtαt , and we have ϵ θ (x t , t) to estimate -σ t ∇ x logp t (x t ), then derive the relationship between L DM (θ) and L FM (θ) We can plug (32) into the loss L F M (θ) in Equation ( 30)EQUATION(36)EQUATION(37)EQUATIONRecall thatEQUATIONwe can see that the difference of L DM (θ) and L FM (θ) during training is caused by the weighted function, which leaving to different trajectories and properties.In this section, we will prove that the proposed pseudo corrector has the same local truncation error and global convergence order as Heun's method. The detailed sampling procedure of Heun's method and pseudo corrector are provided in Algorithm 1 and Algorithm 2. In this section, we use x ti to represent the intermediate sampling result at the t i timestep, and use x * ti = x(t i ) to denote the corresponding ground-truth value on the trajectory. In all the proofs in this section, we omit the condition y for simplicity.Assumption B.1. The velocity predictor v θ (x, t) is Lipschitz continous of constant L w.r.t x.Assumption B.2. The velocity predictor v θ (x, t) has at least 2 derivatives d dt v θ (x, t) and d 2 dt 2 v θ (x, t) and the derivatives are continuous.Assumption B.3. h = max 0≤i≤N -1 h i = O(1/N ),where N is the total number of sampling steps.All the above are common in the analysis of the convergence order of fast samplers [22, 23, 42] of diffusion models.We start by studying the local convergence and Heun's method. Considering the updating from t i to t i+1 and assume all previous results are correct (see the definition of local convergence [15] ). The Taylor's expansion of x * ti+1 at t i gives:EQUATION)On the other hand, let xti+1 be the prediction assuming x i is correct, the updating rule of Heun's method shows:EQUATIONEQUATIONEQUATIONTherefore, the local truncation error can be computed by:EQUATIONwhich indicates that Heun's method has 2 order of accuracy.It is also noted that the local truncation error of the predictor step (which is the same as Euler's method) can be similarly derived by:EQUATION)For pseudo corrector, the analysis of local convergence is the same since we need to assume all previous results (including the x ti and d i ), which means the local truncation error of pseudo corrector is the same as the Heun's method.Global convergence for Heun's method. When analyzing global convergence, we need to take into account both the local truncation error and the effects of the error of previous results. According to the Lipschitz condition, we have:EQUATIONandEQUATIONCombining the above two inequalities together, we haveEQUATIONwhereC 3 = hLC2 2 + C 1 .Note that ∥x * t0 -x t0 ∥ = 0 (their is no error at the beginning of the sampling), it can be easily derived thatEQUATIONTherefore, we have proven that Heun's method have 2 order of global convergence.Global convergence for pseudo corrector. The only difference between pseudo corrector and Heun's method is how d i is obtained. Pseudo corrector reuse the d i from the last sampling step rather than re-compute it as in Heun's method. As a result, d i used in pseudo corrector is computed on xti rather than x ti , which will lead to another error term when analyzing the global convergence.Concretely, the global error of pseudo corrector can be computed by:EQUATIONFor the sake of simplicity, let ∆i = ∥x * ti -xti ∥ and ∆ i = ∥x * ti -x ti ∥. Therefore, the above formulas becomes:EQUATIONEQUATIONBy calculating (52)×hL+(53) we have:EQUATIONNote that ∆ 0 + hL ∆0 = 0. Let ∆ ′ i = ∆ i + hL ∆i , we haveEQUATIONSimilar to the derivation of (49), we can derive thatEQUATIONwhich indicates thatEQUATION)Therefore we have ∆ N ≤ C 9 h 2 , and thus the global convergence of pseudo corrector is 2-order.Table 6 : Ablation of the number of the velocity refiners. We change the number of velocity refiners and compare the sampling quality of each configuration. We find there exists a optimal number of velocity refiners to achieve the lowest FID. Class-conditional image generation. We use the SiT-XL-2 [24] as our base model to perform the experiments on class-conditional image generation. We use a single block of SiT-XL-2 as the Velocity Refiner. We double the input channel from 4 to 8 to take the previous velocity as input. The resulting velocity refiner only contains 29M parameters, about 4.3% of the original SiT-XL-2(675M). We use ImageNet-1K [6]2 to train our velocity model. We used AdamW [21] optimizer for all models. We use a constant learning rate of 5 × 10 -5 and a batch size of 18 on a single A800 GPU. We used a random horizontal flip with a probability of 0.5 in data augmentation. We did not tune the learning rates, decay/warm-up schedules, AdamW parameters, or use any extra data augmentation during training. Our velocity refiner (for SiT-XL-2) trains at approximately 4.44 steps/sec on an A800 GPU, and converges in 30,000 steps, which takes about 2 hours.Text-to-image generation. We use the 2-RF in InstaFlow [20] as our base model to perform the experiments on text-to-image generation. Since the architecture of the original velocity predictor in [20] is a U-Net [30] , we cannot directly use a single block of it as the velocity refiner as we do for SiT [24] . Instead, we simply reduce the number of channels in each block from [320, 640, 1280, 1280] to [160, 160, 320, 320] and reduce the number of layers in each block from 2 to 1. We also double the input channel from 4 to 8 to take the previous velocity as input. The resulting velocity refiner only contains 43.5M parameters, about 5% of the original U-Net (860M). We use a subset of LAION [34] 3 containing only 50K images to train our velocity model. We use AdamW [21] optimizer with a learning rate of 2e-5 and weight decay of 0.0. We adopt a batch size of 16 and set the warming-up steps as 100. We also use a gradient clipping of 0.01 to stabilize training. We train our model on a single A800 GPU for 10K iterations, which takes about 5.5 hours.Implementation of extension tasks. We have demonstrated our FlowTurbo is also suitable for extension tasks due to the multi-step nature of our framework in Section 4.4. For image inpainting, we adopt the inpainting pipeline in diffusion models 4 , where we merge the noise latent and the generated latent at a specific timestep by the input mask. For object removal, we first use a Grounded-SAM 5 to generate the mask and perform similar image inpainting pipeline. For image editing, we adopt the SDEdit [25] which first adds noise to the original image and use it as an intermediate result to continue the sampling.In this section, we provide more analysis through both quantitative results and qualitative results. Ablation of the number of the velocity refiners. In Table 6 , we investigate how to choose the number of velocity refiners to get a better sampling quality. We adopt two basic configurations of H 7 R 10 and H 5 R 10 , and vary the number of velocity refiners from 3 to 6. We find that the FID will first decrease and then increase when N R becomes larger, and there exists an optimal N R = 5 where we reach the lowest FID. These results indicate that we can always tune this hyper-parameter to expect a better result.More comparisons on text-to-image generation. In Table Table 7 , we compare the sampling quality and speed of FLowTurbo with state-of-the-art diffusion models on text-to-image generation. For all the diffusion models, we adopt a 15-step DPM-Solver++ [23] as the default sampler. The FLOPs reported also take the multi-step sampling into account. Our results show that our FlowTurbo can achieve the lowest FID and inference latency.To better illustrate the sampling quality of our FlowTurbo, we provide more qualitative results on both class-conditional image generation and text-to-image generation.Class-conditional image generation. We use SiT-XL [24] as our flow-based model for classconditional image generation. In Figure 5 , we provide random samples from FlowTurbo of the sample config H 8 P 9 R 5 , which inference at 100 ms/img. We also demonstrate the sampling quality trade-offs in Figure 6 , we compare the sampling quality of two different configurations H 1 P 5 R 3 (38 ms / img) and H 8 P 9 R 5 (100 ms / img). We generate the images from the same initial noise for better comparisons. Our result demonstrates that our FlowTurbo can achieve real-time image generation, and the sampling quality can be further improved with more computational budgets.Text-to-image generation. We adopt Lumina-Next-T2I [9] to achieve text-to-image generation. We compare the sampling quality and speed of Heun's method and our FlowTurbo in Figure 7 . We find that FlowTurbo can consistently generate images with better quality and more visual details, while requiring less inference time.Our code is implemented in PyTorch6 . We use the codebase of [24] to conduct experiments. The code is available at https://github.com/shiml20/FlowTurbo. NeurIPS Paper ChecklistLicense: Custom (research, non-commercial)License: Creative Common CC-BY4.0 4 https://huggingface.co/docs/diffusers/en/using-diffusers/inpaint 5 https://github.com/IDEA-Research/Grounded-Segment-Anythinghttps://pytorch.org 


Decision: The paper proposes several improvements to sampling of flow-based models, including lightweight velocity refiner and "pseudo corrector". These together improve the quality-speed tradeoff of sampling, resulting in SOTA quality/speed frontier.

The paper is clearly written, the proposed modifications make sense, the experiments are thorough, and the results are good. All reviewers recommend acceptance. I concur and recommend acceptance too.

Reviews: review: paper_summary: paper_summary: The paper presents a new approach to accelerate the sampling process in flow-based generative models. Unlike diffusion models, flow-based models, which are based on learning velocity fields through flow-matching, have not seen extensive development in efficient sampling techniques. The authors introduce FlowTurbo, a framework that utilizes a lightweight velocity refiner to estimate velocity during sampling, significantly reducing computation time while maintaining or improving image quality. Additionally, the paper proposes pseudo corrector and sample-aware compilation techniques to further enhance sampling efficiency. Experimental results demonstrate that FlowTurbo achieves substantial speedups and high-quality results in both class-conditional and text-to-image generation tasks.
strength_weakness: strength_weakness: 1. **Limited Comparison**: There are many efficient diffusion distillation methods, e.g., consistency distillation [1,2], consistent trajectory model [3], distribution matching distillation [4], adversarial approaches [5,6] (which also distilled using model trained with rectified flow). However, the paper has not been compared with any of those, which makes the comparison incomplete. In order to describe the technical advantage of using lightweight refiner, the author should compare with full fine-tuning (i.e., distillation) or with parameter efficient fine-tuning, e.g., LoRA. Thus, Table 2 should be filled with various baselines in order to precisely state ‘comparison with the state-of-the-arts’. 
2. **Complexity of implementation**: In addition, while the method seems sound, the method is somewhat complicated to use all Heun’s method, corrector step, etc. This might be an obstacle to scale this method to a large-scale.

**References**\
[1] Song, Yang, et al. "Consistency models." arXiv preprint arXiv:2303.01469 (2023). \
[2] Song, Yang, and Prafulla Dhariwal. "Improved techniques for training consistency models." arXiv preprint arXiv:2310.14189 (2023).\
[3] Kim, Dongjun, et al. "Consistency trajectory models: Learning probability flow ode trajectory of diffusion." arXiv preprint arXiv:2310.02279 (2023).\
[4] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\
[5] Sauer, Axel, et al. "Adversarial diffusion distillation." arXiv preprint arXiv:2311.17042 (2023).\
[6] Sauer, Axel, et al. "Fast high-resolution image synthesis with latent adversarial diffusion distillation." arXiv preprint arXiv:2403.12015 (2024).
questions: questions: My major concern is that the paper considers improving the sampling efficiency of flow-based models which is good, but lacks comparison with other fast-sampling diffusion models. Given the theoretical or practical similarities between diffusion models and flow matching models, I believe the previous methods can be also applied to flow matching models. In that sense, there is a large gap in terms of performance, as previous works can sample within only 2-4 steps, e.g., consistency models. In order to elaborate the author’s claim that the velocity refiner is efficient and as good as those methods, the author should provide some additional comparison with those methods.
limitations: limitations: The authors mentioned some limitations that those method could not be used for diffusion models, yet it could be in a better version if they demonstrate with detailed analysis.
paper_summary: paper_summary: The authors propose FlowTurbo, a method to adapt pre-trained flow-based generative models for faster sampling. The method is based on the observation that the predicted velocity field is quite stable throughout the integration time and hence at each timestamp only small refinement from the previous step is required. Therefore, the authors propose to train a lightweight refinement network and use it at certain timestamps instead of the original velocity field predictor. In addition to the refinement network the paper also suggests to speed up the Heun's integration method by reusing the velocities predicted at previous corrector steps. All the sampling blocks are compiled for further speed up. As a result, FlowTurbo demonstrates great trade-off between speed and quality compared to prior work and achieves sota results on certain tasks.
strength_weakness: strength_weakness: 1) **Prior work**: The paper does a good job with contextualizing it relative to the diffusion-based models. However, references to many flow-based methods are missing, which include:

    - Models that leverage flow matching for generative process [2, 4, 5, 6, 7, 8, 9].

    - And (which is more connected to the topic of the paper) the methods that try to speed up the sampling for flow-based models [10, 11, 12, 13].

2) **Method**: Despite the good results, it seems that it is not trivial how to choose the exact configuration of the sampling blocks, i.e. the order and the number of different steps. From the first sight the configurations presented in the paper are quite random. At least some intuition on how to setup those is required. Otherwise, if the configuration needs to be tuned for every other dataset, the method is not that straight-forward to apply.

3) **Evaluation**: 
    - The tables would be simpler to read if NFEs (number of function evaluations) were also included as a separate column.
    - Is the step of the baseline method also compiled in tables 1a and 1b? If not, it would be nice to add this experiment, in addition to the ablations, to see the effect of other parts.

[1] Watson et. al, Novel view synthesis with diffusion models, ICLR 2023

[2] Davtyan et. al, Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV 2023

[4] Hu et. al, Motion Flow Matching for Human Motion Synthesis and Editing, arxiv 2023

[5] Fischer et. al, Boosting Latent Diffusion with Flow Matching, arxiv 2023

[6] Davtyan et. al, Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation, AAAI 2024

[7] Davtyan et. al, Enabling Visual Composition and Animation in Unsupervised Video Generation, arxiv 2024

[8] Gui et. al, Depthfm: Fast monocular depth estimation with flow matching, arxiv 2024

[9] Hu et. al, Flow Matching for Conditional Text Generation in a Few Sampling Steps, arxiv 2024

[10] Shaul et. al, Bespoke Solvers for Generative Flow Models, arxiv 2023, ICLR 2024

[11] Lee et. al, Minimizing Trajectory Curvature of ODE-based Generative Models, ICML 2023

[12] Pooladian et. al, Multisample flow matching: Straightening flows with minibatch couplings, ICML 2023.

[13] Tong et. al, Improving and generalizing flow-based generative models with minibatch optimal transport, TMLR 2023.

[14] Lipman et. al, Flow matching for generative modeling, ICLR 2023.
questions: questions: 1) In the paper the order of blocks $H_{N_H}P_{N_P}R_{N_R}$ is fixed, and only the number of the blocks $N_H$, $N_P$ and $N_R$ changes. What is the reason behind this? Have you tried other schemes, e.g. sequentially using multiple blocks of type $H_{N_H}P_{N_P}R_{N_R}$?

2) A rather nitpicky comment: Multiple times throughout the paper it is mentioned that the linear interpolation corresponds to the optimal transport from the noise distribution to the data distribution. It is not precisely correct, as the linear interpolation corresponds to the optimal transport from the noise distribution to the distribution concentrated around a particular data point [14]. Methods that attempt to steer flow-matching towards optimal transport between noise and data involve additional tricks [12, 13].

Some typos:
1) Line 137: "interpolation between $x_0$ and $\cancel{v}$ $\epsilon$".
2) Line 170: "simulate the $\cancel{x_t}$  $x_{t_i}$".
3) Eq. 15: $\cancel{x_i \leftarrow x_{i-1}}$ $x_{t_i} \leftarrow x_{t_{i - 1}}$
limitations: limitations: The authors have adequately addressed the limitations and the potential negative societal impact of their work.
paper_summary: paper_summary: This paper explores enhancing flow-based generative models for image generation by accelerating the sampling process while maintaining or improving image quality. The key contribution, FlowTurbo, is a new framework that introduces a lightweight velocity refiner to adjust velocity predictions during sampling, reducing computational costs. Additionally, techniques like pseudo correctors and sample-aware compilation further optimize the sampling speed. The results show significant acceleration in image generation tasks with improved FID scores, establishing a new benchmark for real-time image generation.
strength_weakness: strength_weakness: - The design of three types of blocks in sampling is somewhat confusing to me. Why do you have to use these types of blocks and arrange them in this order? Do you have to empirically decide the order and numbers of these blocks for each task/domain/model?
- The visualization in Figure 1 is interesting. However, visualizing the discretization error or curvature [1] during sampling seems to be a more suitable and accurate method.
- Some important related works [1,2], tailored for flow models, should be discussed and compared.

[1] Nguyen, Bao, Binh Nguyen, and Viet Anh Nguyen. "Bellman optimal step-size straightening of flow-matching models." *arXiv preprint arXiv:2312.16414* (2023).

[2] Shaul, Neta, et al. "Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models." *arXiv preprint arXiv:2403.01329* (2024).
questions: questions: If I understand correctly, the proposed method can be applied to diffusion models as well. (Although Figure 1 illustrates flow models may be more suitable.) Have you tested the results on diffusion models like SD or PixArt?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces FlowTurbo, a framework to accelerate flow-based generative models. The key contributions are:
1) a lightweight velocity refiner to estimate the offset of velocity efficiently during sampling;
2) a pseudo corrector to reduce the number of model evaluations while keeping the original second-order convergence;
3) sample aware compilation to compile model evaluations, sampling steps and cfg into a static graph for extra speedup;

Extensive experiments with both class-conditioned and text-to-image generation show significant acceleration (30-58%). Notably, on ImageNet 256x256 generations, FlowTurbo achieves the fastest sampling speed (38 ms/image) with FID 3.93. Moreover, due to the compatibility with multi-step sampling, it can also be applied to other tasks including image editing, inpainting, etc.
strength_weakness: strength_weakness: For text-to-image generation, insta-flow is adopted as the teacher model, which is built on top of sd1.5. It would be great to experiment with other flow-based models that are natively trained with flow matching, for example, sd3. 
It would also be interesting to see how FlowTurbo performs when the model size changes. For example, SiT-XL vs SiT-L. 
Finally, one missing ablation is the architecture of refiner. how to design a refiner for different architectures?
questions: questions: Can this method also be applied to video-based generation models? The inference is much slower for video generation compared with image generation. it would be great to see if there is a unified method for both image and video generation.
limitations: limitations: The proposed method is limited to flow-based generation models.

Comments: title: Thanks for your time and postive feedback
Thanks for your time and postive feedback
Response to the rebuttal
Thanks for raising your score and providing valuable feedback
Thanks for raising your score and providing valuable feedback
comment: We greatly appreciate your response and valuable suggestions, which will improve the quality and impact of our paper. We will incorporate your feedback and update our paper accordingly in the revision.
Thanks the authors for providing the detailed rebuttal and response to my questions. I will maintain my score.
We greatly appreciate your response and valuable suggestions, which would improve the quality and comprehensiveness of our paper. We will update our paper accordingly in the revision.
Dear authors,

Thank you for your time and valuable additional clarifications in the rebuttal.
I will keep my original positive rating.

Best regards,
Reviewer
We are glad to know that most of your concerns have been resolved. FlowTurbo is motivated by the visualization of the sampling trajectory (Figure 1), and leverages the unique property of flow-based models which have $v_t$ as a ''stable value''. As for applying FlowTurbo for pretrained diffusion models,  we think the core idea is to perform some invertible transformation $f$ of the intermediate results $x_t$ such that $f(x_t)$ is a ''stable value'', and learn a lightweight refiner to regress the offset of $f(x_t)$. We will leave this for future work to make our FlowTurbo a more general framework for the acceleration of both diffusion and flow-based models.
Thank you for the rebuttal. 

It is interesting that the proposed method does not applies to diffusion model, even when changed to velocity prediction. So if we want to apply FlowTurbo for pretrained diffusion model, we have to change to flow-based model as InstaFlow did, and then apply FlowTurbo. Elaborating on this point would make the paper more valuable. 

I appreciate the authors' efforts on new experimental results for recent SOTA flow-based models. Most of my concerns were resolved and I update my rating to 5.
Thanks for raising your score and providing valuable feedback. We are glad to know that most of your concerns have been resolved. Regarding the design of our method such as the types of sampling blocks, we will provide more detailed discussions and analyses through both visualizations and experiments (as we have shown in the rebuttal) in the revised paper.
Thanks for this detailed response from the authors. This rebuttal indeed answers many of my questions. Although I still think introducing three types of blocks in a specific order is not the best solution for speeding up, the authors demonstrate its potential, such as in combination with other advanced methods or applying to more advanced models. So, I will raise my score acoordingly.