Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps

Authors: nikita  starodubcev
mikhail  khoroshikh
artem  babenko
dmitry  baranchuk
yandex  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:07:05

Abstract: Diffusion distillation represents a highly promising direction for achieving faithful text-to-image generation in a few sampling steps. However, despite recent successes, existing distilled models still do not provide the full spectrum of diffusion abilities, such as real image inversion, which enables many precise image manipulation methods. This work aims to enrich distilled text-to-image diffusion models with the ability to effectively encode real images into their latent space. To this end, we introduce invertible Consistency Distillation (iCD), a generalized consistency distillation framework that facilitates both high-quality image synthesis and accurate image encoding in only 3-4 inference steps. Though the inversion problem for text-to-image diffusion models gets exacerbated by high classifier-free guidance scales, we notice that dynamic guidance significantly reduces reconstruction errors without noticeable degradation in generation performance. As a result, we demonstrate that iCD equipped with dynamic guidance may serve as a highly effective tool for zero-shot text-guided image editing, competing with more expensive state-of-the-art alternatives.

Main paper: 
 introduction: 
 Recently, text-to-image diffusion models [1, 2, 3, 4, 5, 6] have become a dominant paradigm in image generation based on user-provided textual prompts. The exceptional quality of these models makes them a valuable tool for graphics editors, especially for various image manipulation tasks [7, 8, 9] . In practice, however, the applicability of diffusion models is often hindered by their slow inference, which stems from a sequential sampling procedure, gradually recovering images from pure noise.To speed-up the inference, many recent works aim to reduce the number of diffusion steps via diffusion distillation [10, 11, 12, 13, 14, 15, 16, 17] that has provided significant progress in highquality generation in 1-4 steps and has already been successfully scaled to the state-of-the-art textto-image diffusion models [18, 19, 20, 21, 22, 23, 24] . Though the existing distillation approaches still often trade either mode coverage or image quality for few-step inference, the proposed models can already be feasible for practical applications, such as text-driven image editing [25, 26, 27] .The most effective diffusion-based editing methods typically require encoding real images into the latent space of a diffusion model. For "undistilled" models, this encoding is possible by virtue of the connection of diffusion modeling [28] with denoising score matching [29] through SDE and probability flow ODE (PF ODE) [30] . The ODE perspective of diffusion models reveals their reversibility, i.e., the ability to encode a real image into the model latent space and closely reconstruct it with minimal changes. This ability is successfully exploited in various applications, such as text-driven image editing [31, 32, 33] , domain translation [34, 9] , style transfer [35] .Nevertheless, it remains unclear if distilled models can be enriched with such reversibility since existing diffusion distillation methods primarily focus on achieving efficient generation. This work positively answers this question by proposing invertible Consistency Distillation (iCD), a generalized consistency modeling framework [10, 12, 13] enabling both high-quality image generation and accurate inversion in a few sampling steps.In practice, text-to-image models leverage classifier-free guidance (CFG) [36] , which is crucial for high-fidelity text-to-image generation [1, 3] and text-guided editing [32, 33] . However, the guided diffusion processes yield significant challenges for inversion-based editing methods [32] . Previous approaches [32, 37, 38, 39, 40, 41, 42, 27, 25, 26, 43, 44] have extensively addressed these challenges but often necessitate high computational budget to achieve both strong image manipulations and faithful content preservation. While some of these techniques are applicable to the distilled models [25, 26, 27] , they still dilute the primary advantage of distilled diffusion models: efficient inference.One of the main ingredients of the iCD framework is how it operates with guided diffusion processes. Recently, dynamic guidance has been proposed to improve distribution diversity without noticeable loss in image quality [45, 46] . The key idea is to deactivate CFG for high diffusion noise levels to stimulate exploration at earlier sampling steps. In this work, we notice that dynamic CFG can facilitate image inversion while preserving the editability of the text-to-image diffusion models. Notably, dynamic CFG yields no computational overhead, entirely leveraging the efficiency gains from diffusion distillation. In our experiments, we demonstrate that invertible distilled models equipped with dynamic guidance are a highly effective inversion-based image editing tool.To sum up, our contributions can be formulated as follows:• We propose a generalized consistency distillation framework, invertible Consistency Distillation (iCD), enabling both high-fidelity text-to-image generation and accurate image encoding in around 3-4 sampling steps.• We investigate dynamic classifier-free guidance in the context of image inversion and textguided editing. We demonstrate that it preserves editability of the text-to-image diffusion models while significantly increasing the inversion quality for free.• We apply iCD to large-scale text-to-image models such Stable Diffusion 1.5 [4] and XL [1] and extensively evaluate them for image editing problems. According to automated and human studies, we confirm that iCD unlocks faithful text-guided image editing for 6-8 steps and is comparable to state-of-the-art text-driven image manipulation methods while being multiple times faster. 
background: 
 Diffusion probabilistic models DPMs [29, 28, 47] are a class of generative models producing samples from a simple, typically standard normal, distribution by solving the underlying Probability Flow ODE [30, 48] , involving iterative score function estimation. DPMs are trained to approximate the score function and employ dedicated diffusion ODE solvers [49, 50, 48] for sampling. DDIM [49] is a simple yet effective solver, widely used in text-to-image models and operating in around 50 steps.A single DDIM step from x t to x s can be formulated as follows:EQUATIONwhere α s , α t are defined according to the diffusion schedule [47] , andϵ w θ (x t , t, c) = ϵ θ (x t , t, ⊘) + w(ϵ θ (x t , t, c) -ϵ θ (x t , t, ⊘)) is a linear combination of conditional and unconditional noise predictions called as Classifier-Free Guidance (CFG) [51] , used to improve the image quality and context alignment in conditional generation. In the following, we omit the condition c for simplicity.Due to reversibility, the PF ODE can be solved in both directions: encoding data into the noise space and decoding it back without additional optimization procedures. We refer to this process as inversion [32] where encoding and decoding correspond to forward and reverse processes, respectively.Consistency Distillation CD [10, 12, 13] is the recent state-of-the-art diffusion distillation approach for few-step image generation, which learns to integrate the PF ODE induced with a pretrained diffusion model. In more detail, the model f θ is trained to satisfy the self-consistency property:EQUATIONwhere t n ∈ {t 0 , ..., t N } is a discrete time step, d(•, •) denotes a distance function and x w tn-1 is obtained with a single step of the DDIM solver from t n to t n-1 using the teacher diffusion model. The optimum of ( 2) is defined by the boundary condition, f θ (x t0 , t 0 ) = x t0 . Therefore, consistency models (CMs) learn the transition from any trajectory point to the starting one: f θ (x tn , t n ) = x t0 , ∀ t n ∈ {t 0 , ..., t N }. Consequently, CMs imply a single step generation. However, approximating the entire trajectory using only one step remains highly challenging, leading to unsatisfactory results in practice. To address this, [10] proposes stochastic multistep consistency sampling that iteratively predicts x t0 using f θ and goes back to the intermediate points using the forward diffusion process.The competitive performance of consistency models has stimulated their rapid adoption for text-toimage generation [52, 18, 53] . Nevertheless, we believe that CMs have not yet fully realized their potential in downstream applications, where DPMs excel. One of the reasons is that, unlike DPMs, CMs do not support the inversion process. This work aims to unlock this ability for CMs. Dynamic guidance State-of-the-art text-toimage models employ large CFG scales to achieve high image quality and textual alignment. However, it often leads to the reduced diversity of generated images. To address this, dynamic classifier-free guidance [45, 46, 54] has recently been proposed to improve distribution diversity without noticeable loss in generative performance. CADS [45] gradually increases the guidance scale from zero to the initial high value over the sampling process, Figure 2a . Alternatively, [46] proposes deactivating the guidance for low and high noise levels and using it only on the middle time step interval, Figure 2b . Both strategies suggest that the unguided process at high noise levels is responsible for better distribution diversity without compromising sampling quality. In addition, the authors [46] demonstrate that guidance at low noise levels has a minor effect on the performance and can be omitted to avoid extra model evaluations for guidance calculation. Both dynamic techniques are controlled by two hyperparameters: τ 1 and τ 2 , which are responsible for the value of dynamic CFG w(t). In our work, we focus on the CADS formulation. This section introduces the invertible Consistency Distillation (iCD) framework, which comprises forward and reverse consistency models. First, we formulate the forward CD procedure that encodes images into latent noise. Then, we describe multi-boundary generalization of iCD to enable deterministic multistep inversion. Finally, we investigate the dynamic guidance technique from the inversion perspective.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: the limitations are discussed in the supplementary material. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
forward consistency distillation: 
 Forward Consistency Distillation (fCD) works in the opposite way to CD. That is, it aims to map any point on the PF ODE trajectory to the latent noise (the last trajectory point).The transition from CD to the forward counterpart is quite straightforward: the only thing that should be modified is the boundary condition.Precisely, the forward consistency model is constrained to be an identity function for the last trajectory point: f θ (x t N , t N ) = x t N . Thus, fCD inherits the same consistency distillation loss (2) without incurring extra training costs. This way, the distilled model can transform any trajectory point to the last one:f θ (x tn , t n ) = x t N , ∀ t n .To perform inversion, first, fCD encodes an image into noise and then CD decodes it back. The comparison between CD and fCD is shown in Figure 3a .In practice, the encoding with fCD faces two challenges. Firstly, like in CD, a single-step prediction with fCD can be highly inaccurate. However, this cannot be easily addressed since the multistep consistency sampling [10] is not applicable to fCD. Concretely, intermediate points cannot be obtained from the latent noise using the forward diffusion process. Secondly, even if fCD is accurate, the multistep sampling is not suitable for decoding, as its stochastic nature prevents the reconstruction of real images. So, to improve the prediction accuracy of fCD and reduce the reconstruction error of CD, it is necessary to formulate a deterministic multistep procedure for both models.Recent approaches [53, 13] generalize the CD framework to a multistep regime and allow approximating arbitrary trajectory intervals in the reverse direction. However, these methods focus solely on the generation quality, without supporting the inversion. Thus, inspired by these works, we propose a multi-boundary CD, that unlocks deterministic multistep inversion with the distilled models and carries similar training costs as the classical CD methods.Specifically, we divide the solution interval, {t 0 , ..., t N }, into m segments and perform the distillation on each of these segments separately. This way, we obtain a set of single-step consistency models operating on different intervals and boundary points. This formulation is valid for both CD and fCD and hence can enable deterministic multistep inversion. We provide an illustration of 2-boundary CD and fCD in Figure 3b . We denote the multi-boundary reverse and forward models as CD m and fCD m .Formally, we consider CD m and fCD m using the following parametrization, inspired by [53, 13] .EQUATIONwhere s m t is the boundary time step depending on the number of boundaries, m, and the current time step t. For instance, let m = 1, then s 1 t = t 0 for CD 1 and s 1 t = t N for fCD 1 . Note that we learn a single model, the multistep sampling is achieved by varying s m t during inference. The training objective remains the same as (2), avoiding additional training costs compared to CD. The only limitation is that the number of segments and the corresponding boundary time steps must be set before the training.We train fCD m and CD m separately, initializing both with the same teacher model. We use the same loss with a difference only in boundary time steps. However, a notable difference is the CFG scale, w. For CD m , we preliminarily embed the model on guidance, following [20] , to use various w during sampling and avoid extra model evaluations. For fCD m , we consider an unguided model with a constant w = 1. The reason is that the guided encoding (w > 1) leads to out-of-distribution latent noise [32] , and as a result, to poor image reconstruction. We confirm this intuition in Section 3.4. Finally, we find that m=3-4 provides competitive generation and inversion quality for large-scale text-to-image models [4, 1] . Preservation losses The procedure described above already provides decent inversion quality but still does not match the teacher inversion performance. To reduce the gap between them, we propose the forward and reverse preservation losses aimed at making CD m and fCD m more consistent with each other and improve the inversion accuracy. These losses can additionally be turned on during training. Below, we denote the parameters of CD m , fCD m as θ + , θ -, respectively.The forward preservation loss modifies only fCD m and is described as follows:EQUATIONFor simplicity, we omit some notation. In a nutshell, we sample a noisy image x s m t for a boundary time step s m t , then make a prediction using CD m and force fCD m to predict the same x s m t . This approach encourages CD m and fCD m to be consistent with each other.The reverse preservation loss provides the same intuition but with a difference in the optimized model (CD m instead of fCD m ) and prediction sequence. That is, we first make a prediction using fCD m and then use CD m . We denote it as L r (θ -, θ + ). In our experiments, we calculate the preservation losses only for the unguided reverse process (w = 1).Putting it all together We present our final pipeline for the case where fCD m and CD m are trained jointly starting from the pretrained diffusion model. However, it is possible to learn them by one or take an already pretrained consistency model and learn the rest one. The final objective consists of two consistency losses with the proposed multi-boundary modification and two preservation losses:EQUATIONIn this way, the proposed approach can compete with the state-of-the-art inversion methods using heavyweight diffusion models. We present technical details about the training in Appendix A.As previously discussed, dynamic guidance [45, 46] provides promising results for both faithful and diverse text-to-image generation. In this work, we reveal that dynamic CFG is also an effective technique for improving inversion accuracy as shown in Figure 4b . Below, we delve into the questions when and why dynamic guidance might facilitate image inversion while preserving the generative performance. To answer these questions, we conduct experiments using Stable Diffusion 1.5 with DDIM solver for 50 steps and maximum CFG scale set to 8.0.We start with the dynamic CFG analysis at the decoding stage using the unguided encoding process following the prior work [32] . First, we wonder at which time steps the guidance has the most significant impact on reconstruction quality. To this end, we evaluate MSE between real and reconstructed images for different CFG turn-on thresholds T. If t > T, we set w = 1.0, otherwise, the CFG scale is set to its initial value 8.0. In Figure 4a , we observe an exponential decrease in reconstruction error, implying that the absence of CFG at higher noise levels is essential for achieving more accurate inversion. Figure 4b confirms this intuition qualitatively. These results are consistent with [45, 46] , which also suggest turning off the guidance at high noise levels but motivating this from the perspective of diversity improvement.Then, we investigate the influence of various τ 1 , τ 2 from the CADS dynamic (Figure 2a ) on the inversion and generation performance. We aim to identify an operating point providing both strong Table 1 : FID-5k for SD1.5 starting from the noise latents obtained using different encoding strategies, and NLL for these latents. Though encoding with dynamic CFG produces consistently more plausible latents than constant CFG, the unguided encoding remains preferable.generation performance and faithful image inversion. Thus, we evaluate generation performance using the ImageReward [55] (IR) on top of randomly generated samples for 1000 COCO2014 prompts [56] . The inversion accuracy is estimated in terms of MSE between original and reconstructed samples. Figure 5a presents the results for varying τ 1 and τ 2 . It can be seen that several points for τ 1 ≥ 0.7 offer slightly lower text-to-image performance but exhibit significantly better reconstruction quality compared to the constant CFG scale, 8.0. Moreover, we notice that the settings where τ 1 = τ 2 perform similarly to those where τ 1 < τ 2 . Consequently, in all our experiments, we consider a single τ representing the case where τ 1 = τ 2 and use τ = 0.7 and τ = 0.8. This means that CD m follows unguided sampling for t > τ and sets the initial CFG scale for t ≤ τ .Note that the setting with τ =τ 1 =τ 2 corresponds to a step CFG function w(t), which yields a distinct advantage for distilled models. The linearly changing CFG scales are not applicable to the processes with large discretization steps, typical for distilled diffusion models. Therefore, such a CFG schedule needs to be distilled into the model during training, making it less flexible for different generation and editing settings. In contrast, the step CFG function enables dynamic CFG for already pretrained distilled models, operating with different constant CFG scales. Dynamic guidance for encoding Next, we investigate the guidance role in tackling the encoding problem. In Table 1 , we compare noise latents encoded using various guidance strategies. The quality of the noise latents is estimated by evaluating the generation performance starting from these latents with a fixed CFG scale of 8.0. As the performance measure, we calculate FID for 5000 image-text pairs from COCO [56] .We observe that the latents obtained with a consistently high CFG scale exhibit the worst generative performance, indicating their out-of-domain nature. While dynamic guidance produces significantly more plausible latents, it still falls short of the unguided encoding in most cases. To further validate these results, we estimate the negative log-likelihood (NLL) of the encoded latents under different CFG settings in Table 1 (Bottom). NLL is calculated with respect to the standard normal distribution. While NLL decreases for dynamic CFG with lower τ , the encoding without guidance (w=1) provides the highest likelihood value. Therefore, in all our experiments, we maintain w=1 for the encoding and train the forward distilled models (fCD) on the unguided teacher process.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We attach the code in the supplementary material and will release it upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In the following experiments, we apply our approach to text-to-image diffusion models of different scales: SD1.5 [57] and SDXL [1] , and denote them iCD-SD1.5 and iCD-XL, respectively. We provide the training details in Appendix A.Initially, we illustrate the inversion capability of the proposed framework. Then, we consider the text-guided image editing problem and demonstrate that our approach outperforms or is comparable to significantly more expensive baselines.Before diving into the main experiments, we present a few generated samples using iCD-XL for 4 steps in Figure 6 . Additional quantitative and qualitative results are provided in Appendix B. The results confirm that the distilled model demonstrates strong text-to-image generation performance.Here, we analyze the reconstruction capabilities of iCD-SD1.5 under various configurations. Specifically, we explore the contribution of the different pipeline components, such as the number of steps, preservation losses, and dynamic CFG, to inversion performance. Our forward model is run without CFG (w = 1), while for the reverse model, we consider two settings: unguided (w = 1) and guided (w = 8), both of which are important in practice.Configuration To evaluate the inversion quality, we consider 5K images and the corresponding prompts from the MS-COCO dataset [56] . We measure the reconstruction quality using LPIPS [58] , PSNR and cosine distance in the DinoV2 [59] feature space. As for the reference, the teacher inversion with a disabled CFG scale is considered. For the dynamic guidance, we use τ = 0.7. The coefficients for the preservation losses are equal to λ f = 1.5 and λ r = 1.5.The results are presented in Table 2 . First, configurations (A-C) evaluate the number of the forward and inverse models inference steps. We observe that the reconstruction quality improves as the number of steps increases. In our main experiments, we consider 3 and 4 steps.Then, (E-G) examine the preservation losses. In (E), we learn the forward model in the encoder [60] regime using the forward preservation loss only. This experiment reveals that the consistency loss contributes significantly to inversion performance. (F, G) show that both losses improve the inversion, with the latter approaching the quality of the teacher model.Finally, we explore the dynamic CFG and preservation losses under the guided decoding setting (I-K) and compare them to the setting (H), which does not employ any boosting techniques. From the configurations (I, J, K), we can see that all techniques provide significant contribution to the reconstruction quality. In Figure 7 , we visualize their influence on inversion. It can be seen that the dynamic CFG (I) is rather responsible for global object preservation, while the preservation losses (J,K) rather improve fine-grained details. We note that the final configuration (K) provides comparable inversion quality to the unguided process while preserving the editing capabilities due to the activated guidance. More visual examples of inversion and quantitative results are in Appendix C.In this section, we apply the proposed iCD to the text-guided image editing problem. For the SD1.5 model, we use the Prompt-to-Prompt (P2P) method [61] . We vary two hyperparameters: the cross-attention and self-attention steps balancing between editing strength and preservation of the reference image. We also apply our approach to MasaCTRL [62] in Appendix D. For the SDXL model, we follow the ReNoise [25] evaluation setting and just change the source prompt during decoding according to [63] .Metrics We measure editing performance using both automatic metrics and human-study. The former uses two metrics: 1) to estimate the preservation of the reference image, we calculate the cosine distance between images in the DinoV2 feature space; 2) as an editing quality measure, we use the CLIP score between the edited image and the target prompt. For human evaluation, we employ professional assessors who successfully completed assessment tasks. We show them the source and target prompts, reference image and two images produced with the methods under the comparison Benchmarks In our experiments, we consider two benchmarks: PieBench [31] and a manually created COCO evaluation set based on text-paired images from the MS-COCO dataset [56] . PieBench [31] consists of various types of editing, for example, replacement, addition, or deletion. We take 420 examples of realistic images, including all types of editing. COCO focuses solely on the replacement task as one of the most popular among practitioners. This benchmark contains 140 image-text pairs.Configuration To provide the editing with the SD1.5 model, we consider iCD using 4 forward and 4 reverse steps trained with both preservation losses (λ f = 1.5, λ r = 1.5). In Appendix D, we also present the results for iCD using 3 steps, which is not much worse than the 4 step model. We set the hyperparameters of the dynamic CFG to τ = 0.8 and maximum CFG scale to 19.0. Table 3 : Automatic metrics (top) and human evaluation (bottom) for iCD-XL and ReNoise [25] . Figure 10 : Editing examples using XL models.Baselines We compare our approach with four baseline approaches, which provide state-of-the-art editing performance: Null-text Inversion (NTI), Negative-prompt Inversion (NPI) [32] , InfEdit [26] and Edit-friendly DDPM [43] . All methods except InfEdit are diffusion-based approaches using more than 100 steps. Moreover, NTI employs the additional high-cost optimization procedure to improve inversion quality. InfEdit operates with the distilled diffusion model, latent consistency distillation [18] , but utilizing virtual inversion. All methods use P2P and we vary all possible hyperparameter values to find the configurations that provide the best editing-preservation trade-off.Results Figure 9 provides quantitative results for both benchmarks. We observe that the proposed iCD is comparable to the baseline approaches in most cases. Moreover, sometimes it can even outperform them while being multiple times faster. For instance, according to human preference on the COCO benchmark, our approach surpasses the InfEdit, NPI, NTI and Edit-friendly DDPM (50 steps). On the PieBench, it outperforms the InfEdit and Edit-friendly DDPM (50 steps). In addition, we provide qualitative results in Figure 8 , which confirm the competitiveness of the proposed method.Additional visual examples can be found in Appendix D.Notably, the performance of the proposed method is weaker on the PieBench benchmark compared to COCO, according to both automatic and human-based metrics. We attribute this to the increased complexity of editing tasks, which probably require more steps.Configuration We consider the configuration using 4 steps, τ = 0.7 and CFG scale equals 8.0.Baselines We compare our approach to the recently proposed ReNoise [25] , accurately following its guidelines. This method works with both distilled models (LCM-SDXL [18] , SDXL-Turbo [19] ) and original diffusion model, SDXL [1] . However, even for the distilled models, a significant number of steps is required to achieve decent performance.The quantitative and qualitative comparisons are presented in Table 3 and Figure 10 , respectively. According to the human evaluation, iCD-XL outperforms all ReNoise configurations. Based on the automatic evaluation, our approach provides better reference preservation (DinoV2 and CLIP score, I) while maintaining strong editing capabilities, as indicated by the CLIP score (T). We provide more visual examples in Appendix D. 
conclusion: 
 The paper proposes a generalized consistency distillation framework that enables both accurate image inversion and solid generation performance using a few inference steps. Accompanied by the recently proposed dynamic guidance, the distilled models demonstrate highly efficient and accurate image manipulations, making a significant step towards real-time text-driven image editing. First, following [20] , we preliminary distill classifier-free guidance using a conditional embedding added to the time step embedding. The goal is to use different w and apply dynamic techniques during inference after consistency distillation. We perform CFG distillation for the following guidance scales: 1, 8, 12, 16, 20 for SD1.5 and 1, 4, 6, 9, 10, 12, 13, 16, 18, 20 for SDXL. At this stage, the model successfully approximates the guided teacher without hurting its performance. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: To the best of our judgment, we provide the details sufficient to reproduce the results. However, we do not discuss some low-level implementation details that are not critical for understanding, and refer the readers to our code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We provide the confidence intervals for the main experiments and and explain their nature in the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We will provide guidelines for users upon acceptance of a paper to control the use of our models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We plan to release our code and models upon acceptance under the CC-by 4.0 license. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: Human evaluation is performed by professional assessors. They are officially hired, paid competitive salary and informed about potential risks. We include the instruction in the supplementary material. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: Before involving participants in the evaluation, we provide them with detailed instructions addressing potential risks. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Then, we perform multi-boundary consistency distillation using LoRA adapters with a rank of 64 following [52] . We train forward and reverse adapters in parallel starting from the same teacher: CFG distilled SD1.5 or SDXL. For the SD1.5 model, we use a global batch size of 512, and for the SDXL 128. All models converge relatively fast, requiring about 6K iterations with a learning rate 8e-6.We find that the forward and reverse models provide promising generation and inversion quality for 3 or 4 steps. The regularization coefficients for the forward and reverse preservation losses are λ f =1.5 and λ r =1.5, respectively.The iCD-SD1.5 models are trained for ∼36h and the iCD-XL ones for ∼68h on 8 NVIDIA A100 GPUs. We present the training dynamics in terms of FID and reconstruction MSE for iCD-SD1.5 in Figure 11 .For SD1.5 distillation, we use a ∼20M subset of LAION 2B, roughly filtered using CLIP score [64] .For SDXL, we collect ∼7M images with resolution ≥ 1024, also curated to avoid poorly aligned text-image pairs and low quality images.We set the following time steps for our configurations:• 4 steps, τ = 0. We provide the generation performance of our distilled model in Table 4 . The dynamic CFG (τ = 0.8, τ = 0.7) degrades in terms of ImageReward, while improving FID due to the increased diversity [45, 46] . The visual examples are presented in Figures 16, 17 .In Table 5 , we provide comparisons between the 3and 4-step configurations of iCD-SD1.5, which perform similarly. Figure 14 shows the image inversions provided by our approach, NTI and NPI. Figure 12 compares two iCD configurations (3 and 4 steps). We observe that both configurations perform similarly, with a slight preference for the 4-step configuration. We present additional visual results on image editing using the iCD-SD1.5 model in Figure 18 and the iCD-XL model in Figure 19 .Figure 13 provides a few examples of our approach combined with MasaCTRL [62] for non-rigid editing. The results demonstrate that our method can be used with various editing methods.We also present the comparison with SDXL-Turbo using SDEdit [63] in Figure 20 (upper). We observe that SDXL Turbo significantly hurts reference image preservation due to stochasticity. This highlights the importance of accurate image inversion for editing. Moreover, we compare our approach with Intruct-Pix2Pix [7] , Figure 20 (bottom), and observe that it outperforms Intruct-Pix2Pix in terms of both content preservation and editing strength while being training-free.We present the annotation interface for the assessors in Figure 22 . We calculate the confidence interval using bootstrap methods, splitting the human votes into 1, 000 subsets and then averaging the results and calculating the standard deviation.We present some inversion failure cases in Figure 21 . Our method sometimes oversaturates images for high guidance scales and struggles to reconstruct complex details like human faces and handsThe iCD limitations include the requirement to predetermine boundary time steps before distillation, which may restrict the model flexibility. Additionally, the extra preservation losses necessitate increased computational resources during training. Furthermore, the editing method is susceptible to hyperparameter values, leading to inconsistent performance across different prompts. Finally, the quality of the current distillation techniques itself requires significant improvement, as it does not consistently achieve high standards across various scenarios.Our work can significantly enhance tools for artists, designers, and content creators, allowing for more precise and efficient manipulation of images based on textual inputs. This can democratize high-quality digital art creation, making it accessible to those without extensive technical skills. On the other hand, the ability to edit images easily and realistically can be misused to create misleading information or fake images, which can be particularly harmful and potentially influence public opinion and elections. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper extends distilled text-to-image diffusion models to fast editing by introducing invertible consistency distillation, which allows efficient image inversion and synthesis. All reviewers give positive scores after rebuttal. The problem is interesting, the method is novel, and the performance is significant. AC agrees this is a nice contribution on diffusion models and image editing, and thus recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper targets a novel problem of enabling image inversion and editing for models distilled with consistency distillation. 
The authors identify the challenges of applying consistency distillation, which is designed for the denoising process, to the diffusion process and propose multi-boundary consistency distillation that achieves inversion with as few as 3-4 steps. A regularization term is proposed to guarantee the consistency between the forward and backward models. The authors further study dynamic CFG for performance improvements. The effectiveness is validated on SD 1.5 and SDXL models.
strength_weakness: strength_weakness: - While the overall problem is novel, and the solution demonstrates strong performance, almost every individual technique presented in this paper comes from another earlier work and makes the overall contribution less impactful.

- The proposed framework is highly customized toward CD, and might not be able to generalize other DM model acceleration methods such as SD-turbo, DMD, and UFOGen.
questions: questions: The presented evaluations seem to focus primarily on 7 steps. While the performance is impressive compared to other methods, the higher performance reported by fDDIM50 makes me wonder if the proposed method can match the performance with more steps and what the marginal performance would be if we scale up the number of steps. 

Minor suggestion, doesn't affect rating:
The overall quality of this paper can be greatly improved by replacing Figure 1. The current one seems too simple and needs to be more informative. Figure 6 can even be a better alternative for Figure 1.
limitations: limitations: The authors have properly discussed the limitations in Appendix Section E.
paper_summary: paper_summary: This work introduces invertible Consistency Distillation (iCD), which enhances text-to-image diffusion models by enabling effective encoding of real images into latent space. iCD achieves both high-quality image synthesis and accurate image inversion in just 3-4 inference steps.
strength_weakness: strength_weakness: 1. Discussion of Failure Cases: It would be beneficial to include a detailed discussion of failure cases. Understanding the scenarios where the method does not perform optimally can provide valuable insights and guide future research directions.

2. More SDXL Demos Needed: The main paper (Figure 6) showcases demos using SD1.5, which, however, still suffers from generation issues like details of hands and faces. It is better to include more examples using SDXL (which offers improved performance) in the main paper. This would strengthen the evaluation and illustrate the capabilities of the proposed method more effectively.

3. Comparison with SDXL Turbo (Adversarial Distillation): SDXL turbo is known for its excellent performance in one-step generation and can be used for editing with tools like SDedit. A comparative analysis of the editing effectiveness and efficiency between the proposed method and adversarial distillation methods would provide a clearer understanding of the advantages and potential trade-offs, offering a more comprehensive evaluation of the proposed approach.
questions: questions: Please refer to the weaknesses.
limitations: limitations: This work has discussed limitations in Appendix.
paper_summary: paper_summary: This paper extend the idea of consistency distillation to inversion for image editing. By training a separate consistency model where the consistency is enforced at noise space rather than latent space. Additional cycle consistency loss is employed for more accurate inversion.
strength_weakness: strength_weakness: 1. The description of the method is unclear. For example, section 3.1 and 3.2 are the main idea of the method, but it ignores an important component: the data. When training consistency distillation, we use perturbed real data, but since here we need to map to the noise, we need to have a coupling of the image and noise. How to obtain this coupling? Do we need to run inversion with the teacher model on a large dataset to obtain the training data? This is very important component of the method but I didn't find an explanation throughout the paper.

2. The method basically needs two models, by employing CD in forward and backward directions, for fast inversion and generation. Although it is practically useful, I feel like it is a bit ad-hoc. The reason is that, if we enable two models, one for fast generation and the other for fast inversion, then any diffusion acceleration methods can be employed separately for two models. It would be more interesting to study this question: given a distilled diffusion model that can be sampled in few steps, can we use this model to do fast inversion?

3. Related to above point. Two models introduce significantly more parameter overhead, and given that inversion based image editing is only only handle a small subset of image editing tasks, it becomes hard to justify whether it worths to double the parameter.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes distilation for both of forward and reverse path of ODE sampling of diffusion models in order to enable faster editing.
strength_weakness: strength_weakness: 1. Although the method shows great performance in image editing with conditions, it feels that the editing method is limited on Prompt2prompt. Is it still possible to apply the model on non-rigid editing such as MasaCTRL? 

2. It seems that the most important contributions of this model is faster inversion & sampling. Please explicitly compare the inversion&sampling time with other baseline methods and put the table in main paper. 

3. In the experiment part, please compare the model with other training (or fine-tuning) based methods such as instruct-pix2pix or ControlNet. The current experiment only focuses on inversion methods, therefore further experiment would be more helpful for the manuscript.

4. Does the performance degrade if the distillation step decreases? Please give more discussion and experimental results of the case of decreased steps such as single step of 4 steps.  

5. minor comment, The Figure 1 (First figure) seems very redundant for me. I recommend a new figure which contains the brief summary of methodology such as combined version of Figure 3 and Figure 1. The figure in the current version make the paper seems unprofessional. As I like the overall idea, it would greatly improve the readability of the manuscript with changing the first figure.
questions: questions: Already written in the weakness part.
limitations: limitations: Yes

Comments: title: After rebuttal
comment: Thanks authors for the response.

Now I am clear about the data you are using. But I am still not convinced that if it worths to have two models just for fast inversion. I will increase my score to 5 to reflect your clarification.
Thank you for the rebuttal. I have no further concerns. Adding these detailed discussions and comparisons will undoubtedly enhance the quality of the paper.
Dear Reviewer Qhsn,

Given the limited time for discussion, we would appreciate it if you could let us know whether we have fully addressed your concerns and whether there are any other questions we might need to address.
Dear Reviewer QtT7,

Given the limited time for discussion, we would appreciate it if you could let us know whether we have fully addressed your concerns and whether there are any other questions we might need to address.
Dear Reviewer H4g5, 

Given the limited time for discussion, we would appreciate it if you could let us know whether we have fully addressed your concerns and whether there are any other questions we might need to address.
Thank you for your detailed rebuttal. Most of my concerns have been addressed.