Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps
Authors: Nikita Starodubcev
Mikhail Khoroshikh
Artem Babenko
Dmitry Baranchuk
Venue: NeurIPS 2024 poster (2024)
Keywords: text-guided image editing
consistency distillation
diffusion models
image inversion
Primary area: diffusion_based_models
TL;DR: zero shot image editing with consistency distillation

Abstract: Diffusion distillation represents a highly promising direction for achieving faithful text-to-image generation in a few sampling steps. However, despite recent successes, existing distilled models still do not provide the full spectrum of diffusion abilities, such as real image inversion, which enables many precise image manipulation methods. This work aims to enrich distilled text-to-image diffusion models with the ability to effectively encode real images into their latent space. To this end, we introduce invertible Consistency Distillation (iCD), a generalized consistency distillation framework that facilitates both high-quality image synthesis and accurate image encoding in only 3-4 inference steps. Though the inversion problem for text-to-image diffusion models gets exacerbated by high classifier-free guidance scales, we notice that dynamic guidance significantly reduces reconstruction errors without noticeable degradation in generation performance. As a result, we demonstrate that iCD equipped with dynamic guidance may serve as a highly effective tool for zero-shot text-guided image editing, competing with more expensive state-of-the-art alternatives.

Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps
Nikita Starodubcey' 2 Mikhail Khoroshikh'
Artem Babenko'
Dmitry Baranchuk'
Yandex Research
HSE University
httpa://yandex-reeenn
github. io/invertible-ca
Original
Editing (~ 0.9 secs)
dog bear
wearing a hoodie
dog raccoon
blue t-shirt
raising his hand on the beach
Figure 1:
Invertible Consistency Distillation (iCD) enables both fast image editing and strong generation performance in a few model evaluations.
Abstract
Diffusion distillation represents a highly promising direction for achieving faith- ful text-to-image generation in a few sampling steps. However, despite recent successes, existing distilled models still do not provide the full spectrum of dif- fusion abilities, such as real image inversion, which enables many precise image manipulation methods. This work aims enrich distilled text-to-image diffusion models with the ability to effectively encode real images into their latent space. To this end, we introduce invertible Consistency Distillation (iCD), generalized consistency distillation framework that facilitates both high-qualin image synthesis and accurate image encoding in only 3-4 inference steps. Though the inversion problem for text-to-image diffusion models gets exacerbated by high classifier-free guidance scales, we notice that dynamic guidance significantly reduces recon- struction errors without noticeable degradation in generation performance. As a result, we demonstrate that iCD equipped with dynamic guidance may serve as a highly effective tool for zero-shot text-guided image editing, competing with more expensive state-of-the-ar alternatives.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Recently, text-to-image diffusion models 2, 4, 6] have become dominant paradigm in image generation based on user-provided textual prompts. The exceptional quality of these models makes them valuable tool for graphies editors, especially for various image manipulation tasks [7, 8, 9]. In practice, however, the applicability of diffusiin models is often hindered by their slow inference, which stems from a sequential sampling procedure, gradually recovering images from pure noise.
To speed-up the inference, many recent works aim to reduce the number of diffusion steps via diffusion distillation [10, 11. 12, 13. 14, 15, 16, 17] that has provided significant progress in high- quality generation in 1-4 steps and has already been successfully scaled to the state-of-th of-the-ar art text- to-image diffusion models [18, 19, 20, 21, 22, 23, 24]. Though the existing distillation approaches still often trade either mode coverage or image quality for few-step inference, the proposed models already be feasible for practical applications, such text-driven image editing [25, 26, 27].
The most effective diffusion- editing methods typically require encoding real images into the latent space of diffusion model. For "undistilled" models, this encoding is possible by virtue of the connection of diffusion modeling [28] with denoising score matching [29] through SDE and probability flow ODE (PF ODE) [30]. The ODE perspective of diffusion models reveals their reversibility, i.e., the ability to encode real image into the model latent space and closely reconstruct it with minimal changes. This ability is successfully exploited in various applications, such as text-driven image editing [31, 32, 33), domain translation [34, 9]. style transfer [35].
Nevertheless, it remains unclear if distilled models can be enriched with such reversibility since existing diffusion distillation methods primarily focus on achieving efficient generation. This work positively answers this question by proposing invertible Consistency Distillation (iCD), generalized consistency modeling framework [10, 12, 13] enabling both high-quality image generation and accurate inversion in few sampling steps.
In practice, text-to-image models leverage classifier-free guidance (CFG) [36]. which is crucial for high-fidelity text-to-image generation [1, 3] and text-guided editing [32, 33]. However, the editing methods [32]. Previous approaches [32, 37, 38, 39, 40, 41, 42, 27, 25, 26, 43, 44] have extensively addressed these challenges but often necessitate high computational budget to achieve both strong image manipulations and faithful content preservation. While some of these techniques are applicable to the distilled models [25, 26, 27], they still dilute the primary advantage of distilled diffusion models: efficient inference.
guided diffusion processes yield significant challenges for inversion
One of the main ingredients of the iCD framework is how it operates with guided diffusion processes. Recently, dynamic guidance has been proposed to improve distribution diversity without noticeable loss in image quality [45, 46]. The key idea is to deactivate CFG for high diffusion noise levels to stimulate exploration at earlier sampling steps. In this work, we notice that dynamic CFG can facilitate image inversion while preserving the editability of the text-to-image diffusion models. Notably, dynamic CFG yields no computational overhead, entirely leveraging the efficiency gains from diffusion distillation. In our experiments, we demonstrate that invertible distilled models equipped with dynamic guidance are highly effective inversion-based image editing tool.
To sum up, our contributions can be formulated follows:
We propose a generalized consistency distillation framework, invertible Consistency Dis- tillation (iCD), enabling both high-fidelity text-to-image generation and accurate image encoding in around 3-4 sampling steps.
We investigate dynamic classifier-free guidance the context of image inversion and text- guided editing. We demonstrate that it preserves editability of the text-to-image diffusion models while significantly increasing the inversion quality for free.
We apply iCD to large-scale text-to- image models such Stable Diffusion 1.5 [4] and XL [1] and extensively evaluate them for image editing problems. According to automated and human studies, we confirm that iCD unlocks faithful text-guided image editing for 6-8 steps and is comparable to state-of-thh-arr text-driven image manipulation methods while being multiple times faster.

---

2 Background
Difffusion probabillitte models DPMs [29, 28, 47] are a class of generative models producing samples from simple, typically standard normal, distribution by solving the underlying Probability Flow ODE [30, 48], involving iterative score function estimation. DPMs are trained to approximate the score function and employ dedicated diffusion ODE solvers [49, 50, 48] for sampling. DDIM [49] a simple yet effective solver, widely used in text-to-image models and operating in around 50 steps. A single DDIM step from 2t to Bt can be formulated as follows:
x #
DDIM (2;, w,
vt Eg (ar, c) (V1- as
V1 0t az
(I)
04
where ag O2 are defined according to the diffusion schedule [47], and Eă (2 t, (at, t, c) - co (zz t, ()) is a linear combination of conditional and unconditional noise predic- tions called as Classifier-Free Guidance (CFG) [51], used to improve the image quality and context alignment in conditional generation. In the following, we omit the condition c for simplicity.
(z O)
Due to reversibility, the PF ODE can be solved in both directions: encoding data into the noise space and decoding it back without additional optimization procedures. We refer to this process as inver- sion [32] where encoding and decoding correspond to forward and reverse processes, respectively.
Consisteney Distillation CD [10, 12, 13] is the recent state-of-the-ar diffusion distillation approach for few-step image generation, which learns to integrate the PF ODE induced with a pretrained diffusion model. In more detail, the model fe is trained satisfy the self-consistency property:
( = [d( (2t.
Fo +x)) min,
(2)
where to E {to..... (N } is a discrete time step, -(.,-) denotes a distance function and o is obtained with a single step of the DDIM solver from ln to l--1 using the teacher diffusion model. The optimum of (2) is defined by the boundary condition, ( 4n) Btg Therefore, consistency models (CMs) learn the transition from any trajectory point to the starting one: (z (.) ...,
Lt In E (lo:...,NN) (N). J. Consequently, CMs imply a single step generation. However, approximating the entire trajectory using only one step remains highly challenging, leading to unsatisfactory results in practice. To address this, [10] proposes stochastic multistep consistency sampling that iteratively predicts It. using fo and goes back to the intermediate points using the forward diffusion process.
The competitive performance of consistency models has stimulated their rapid adoption for text-to- image generation [52, 18, 53). Nevertheless, we believe that CMs have not yet fully realized their potential in downstream applications, where DPMs excel. One of the reasons is that, unlike DPMs, CMs do not support the inversion process. This work aims to unlock this ability for CMs.
Dynamic guidance Sfate-of-the-art text-to- image models employ large CFG scales to achieve high image quality and textual align- ment. However, it often leads to the reduced diversity of generated images. To address this, dynamic classifier-free guidance [45, 46, 54] has recently been proposed to improve distribution diversity without noticeable loss in generative performance. CADS [45] gradually increases the guidance scale from zero to the initial high value over the sampling process, Figure 2a. Al- ternatively, [46] proposes deactivating the guid- ance for low and high noise levels and using only on the middle time step interval, Figure 2b. Both strategies suggest that the unguided process at high noise levels is responsible for better distribution diversity without compromising sampling quality. In addition, the authors [46] demonstrate that guidance at low noise levels has minor effect on the performance and can be omitted to avoid extra model evaluations for guidance calculation. Both dynamic techniques are controlled by two hyperparameters Tj and 72, which are responsible for the value of dynamic CFG (t). In our work, we focus on the CADS formulation.
w(0:
1.0
1.0
6 1:
(u) CADS dynamic
(b) [45] dynamic
Figure 2: Dynamic CFG strategies.
3

---

3 Method
ODE trajectory
This section introduces the invertible Consis- tency Distillation (iCD) framework, which com- prises forward and reverse consistency models. First, we formulate the forward CD procedure that encodes images into latent noise. Then, we describe multi-boundary generalization of iCD to enable deterministic multistep inversion. Finally, we investigate the dynamic guidance technique from the inversion perspective.
CD"/ fCD boundary points
(a) 1 boundary CD"/ fCD"
3.1 Forward Consistency Distillation
Ex
Forward Consistency Distillation (fCD) works in the opposite way to CD. That is, it aims to map any point on the PF ODE trajectory to the latent noise (the last trajectory point).
(b) 2 boundary CD"/ fCD"
The transition from CD to the forward counter- part is quite straightforward: the only thing that should be modified is the boundary condition. Precisely, the forward consistency model is con- strained be identity function for the last tra- jectory point: fe x (2) = Ity. Thus, fCD inherits the same consistency distillation loss (2)
Figure 3: The proposed invertible Consistency Dis- tillation framework consists of two models: the forward m-boundary model, ICD",, and the re- verse model, CD"" (a) For m model corresponds to CD. More boundary points unlock the deterministic multistep inversion, e.g.. (b) shows the case for m = 2.
the reverse
without incurring extra training costs. This way, the distilled model can transform any trajectory point to the last one: ( (n) = Dey: V bn- To perform inversion, first, fCD encodes an image into noise and then CD decodes it back. The comparison between CD and fCD is shown in Figure 3a.
3.2 Multi-boundary Consistency Distillation
In practice, the encoding with ICD faces two challenges. Firstly, like CD, a single-step prediction with fCD can be highly inaccurate. However, this cannot be easily addressed since the multistep consistency sampling [10] is not applicable to fCD. Concretely, intermediate points cannot be obtained from the latent noise using the forward diffusion process. Secondly, even if fCD is accurate, the multistep sampling is not suitable for decoding, as its stochastic nature prevents the reconstruction of real images. So, to improve the prediction accuracy of fCD and reduce the reconstruction error of CD, is necessary to formulate a deterministic multistep procedure for both models.
Recent approaches [53, 13] generalize the CD framework to multistep regime and allow approxi- mating arbitrary trajectory intervals in the reverse direction. However, these methods focus solely on the generation quality, without supporting the inversion. Thus, inspired by these works, propose a multi- bouudarourr CD, that unlocks deterministic multistep inversion with the distilled models and carries similar training costs as the classical CD methods.
Specifically, we divide the solution interval, (to....,tyy into m segments and perform the distillation on each of these segments separately. This way, we obtain set of single-step consistency models operating on different intervals and boundary points. This formulation is valid for both CD and fCD and hence can enable deterministic multistep inversion. We provide an illustration of 2-boundary CD and fCD in Figure 3b. We denote the multi-boundary reverse and forward models as "D and fCD"..
Formally, we consider CD" and fCD" using the following parametrization, inspired by [53, 13).
E.m f (æx, I, 8 w) DDIM , w),
where st is the boundary time step depending on the number of boundaries, m, and the current time step t. For instance, let m = 1, then s to for CD' and s! = tv for fCD!. Note that we learn a single model, the multistep sampling is achieved by varying 8;" during inference. The training objective remains the same as (2), avoiding additional training costs compared to CD. The only limitation that the number of segments and the corresponding boundary time steps must be set before the training.

---

3.3 Training ICD and CD
We train iCD" and CD separately, initializing both with the same teacher model. We use the same loss with a difference only in boundary time steps. However, notable difference is the CFG scale, w. For CDT, we preliminarily embed the model on guidance, following [20], to use various w during sampling and avoid extra model evaluations. For fCD",, we consider an unguided model with constant w = 1. The reason is that the guided encoding (w > 1) leads to out-of-distribution latent result, to poor image reconstruction. We confirm this intuition in Section 3.4.
noise [32], and Finally, we find that m=3-4 provides competitive generation and inversion quality for large-scale text-to-image models [4, I].
Preservation losses The procedure described above already provides decent inversion quality but still does not match the teacher inversion performance. To reduce the gap between them, we propose the forward and reverse preservation losses aimed at making CD and ICD" more consistent with each other and improve the inversion accuracy. These losses can additionally be turned on during training. Below, we denote the parameters of CD", ICD" as 0 0 respectively.
The forward preservation loss modifies only fCD and is described as follows:
f = E [d (fØ (S (xx.)) )), axr) min,
)),
For simplicity, we omit some notation. In nutshell, sample noisy image x,, for boundary time step s,", then make a prediction using CD and force fCD" to predict the same Tog. This approach encourages CD" and fCD" be consistent with each other.
The reverse preservation loss provides the same intuition but with a difference the optimized model (CD"" instead of fCD"") and prediction sequence. That is, first make prediction using ICD and then use CD" We denote it as (, 0*). In our experiments, we calculate the preservation losses only for the unguided reverse process (w 1).
Putting it all together We present our final pipeline for the case where fCD" and CD are trained jointly starting from the pretrained diffusion model. However, it is possible learn them by one or take an already pretrained consistency model and learn the rest one. The final objective consists of two consistency losses with the proposed multi-boundary modification and two preservation losses:
CeD(Ot ,0) (cD(0 + ,,C ,7)
(5)
In this way, the proposed approach can compete with the state-of-the-arr inversion methods using heavyweight diffusion models. We present technical details about the training in Appendix A.
3.4 Dynamic Classifier- Guidance Facilitates Inversion
As previously discussed, dynamic guidance [45, 46] provides promising results for both faithful and diverse text-to-image generation. In this work, we reveal that dynamic CFG is also an effective technique for improving inversion accuracy as shown in Figure 4b. Below, we delve into the questions when and why dynamic guidance might facilitate image inversion while preserving the generative performance. To answer these questions, conduct experiments using Stable Diffusion 1.5 with DDIM solver for 50 steps and maximum CFG scale set to 8.0.
Dynamic guidance for decoding We start with the dynamic CFG analysis at the decoding stage using the unguided encoding process following the prior work [32]. First, we wonder at which time steps the guidance has the most significant impact reconstruction quality. To this end, we evaluate MSE between real and reconstructed images for different CFG turn-on thresholds T. If > T, we set = 1.0, otherwise, the CFG scale is set to its initial value 8.0. In Figure 4a, observe an exponential decrease in reconstruction error, implying that the absence CFG at higher noise levels is essential for achieving more accurate inversion. Figure 4b confirms this intuition qualitatively. These results are consistent with [45, 46), which also suggest turning off the guidance at high noise levels but motivating this from the perspective of diversity improvement.
Then, we investigate the influence of various TI, T2 from the CADS dynamic (Figure 2a) on the inversion and generation performance. We aim to identify operating point providing both strong
5

---

Original Image
Inversion
T
1.0
Recostractio MSE.
dynamir CFG
0.2
consinnt CFG, 100
0.1
CPG furn-on thrshhods
(a)
(b)
Figure 4: (a) Reconstruction
of the decoding process for different CFG turn-on thresholds. (b) Image inversion examples for different CFG turn-on thresholds T. Guidance at high noise levels
(T = 1.0) drastically degrades the inversion quality.
1.0
Generation
constant CFG. 1.0
0.9
constant CFG, 8.0
Inage/Rewardd f
0.8
-0.3
7t
0.1
0.6
11.5
-0.8
7u
0.4
comstont CDG,
CPG,
-1.3
0.1
0.2
0.3
0.4
0.0
Recoustruction MSE,
a barred owl peeking out from dense free branches
(a)
(b)
Figure 5: (a) Trade-off between generation performance (IR) and reconstruction quality (MSE) pro- vided by different 71, 72- (b) Generation examples for dynamic and constant CFG scales. The points around 72 0.8 provide preferable trade-off between generation and inversion performance.
Enc. No CFG
Enc, d.CFG, 7=0.6
Enc, d.CFG, ==0.8
Enc, CFG
Dec. No CFG
11.0
36.6
63.4
100.5
Dec. d.CFG 706.6
11.4
23.5
67.8
102.6
Dec. d.CFG =08.8
15.0
14.6
52.2
108.5
Dec. CFG
19.0
19.2
19.0
102.1
Latent NIL,
1.401
1.409
1.415
1.428
Table 1: FID-5k for SD1.5 starting from the noise latents obtained using different encoding strategies,
and NLL for these latents. Though encoding with dynamic CFG produces consistently more plausible latents than constant CFG, the unguided encoding remains preferable.
generation performance and faithful image inversion. Thus, evaluate generation performance using the ImageReward [55] (IR) on top randomly generated samples for 1000 COC02014 prompts [56].
The inversion accuracy is estimated in terms of MSE between original and reconstructed samples.
Figure presents the results for varying T and 72. It can be seen that several points for t 2 0.7 offer slightly lower text-to-image performance but exhibit significantly better reconstruction quality compared to the constant CFG scale, 8.0. Morcover. we notice that the settings where Ti = T2 perform similarly to those where Ti T2. Consequently, in all experiments, consider single t representing the case where Ti = T2 and use 7 = 0.7 and 0.8. This means that CD follows unguided sampling for é > and sets the initial CFG scale for t <
Note that the setting with == corresponds to step CFG function w(t), which yields distinct advantage for distilled models. The linearly changing CFG scales are not applicable to the processes with large discretization steps, typical for distilled diffusion models. Therefore, such a CFG schedule needs to be distilled into the model during training, making it less flexible for different generation and editing settings. contrast, the step CFG function enables dynamic CFG for already pretrained distilled models, operating with different constant CFG scales.

---

stops
Teacher. 50 stops
Ours, stops
Teacher, 50 steps
donkey castume grving focture the front of The blackboard has mathematical equations on it.
on airplane faxiing runway
castle made of tortilla chips, river made of salsa. There siny burvitos walking around the castle
A raccson woaring formal clothes, wearing ixphat and holding cane.
The raccoon holding garbage bug. Oll painting the style cubism.
Figure 6: Few examples of text-to-image generation using the iCD-XL model for 4 steps.
Dynamic guidance for encoding Next, we investigate the guidance role in tackling the encoding problem. In Table 1, we compare noise latents encoded using various guidance strategies. The quality of the noise latents is estimated by evaluating the generation performance starting from these latents with fixed CFG scale of 8.0. As the performance measure, calculate FID for 5000 image-text pairs from COCO [56].
We observe that the latents obtained with consistently high CFG scale exhibit the worst generative performance, indicating their out-of-domman nature. While dynamic guidance produces significantly more plausible latents, it still falls short of the unguided encoding in most cases. To further validate these results, we estimate the negative log-likelihood (NLL) of the encoded latents under different CFG settings in Table (Bottom). NLL is calculated with respect the standard normal distribution. While NLL decreases for dynamic CFG with lower 7, the encoding without guidance (w=1) provides the highest likelihood value. Therefore, in all our experiments, we maintain w=1 for the encoding and train the forward distilled models (fCD) on the unguided teacher process.
4 Experiments
In the following experiments, we apply our approach to text-to-image diffusion models of different scales: SD1.5 [57] and SDXL [1], and denote them iCD-SD1.5 and iCD-XL, respectively. We provide the training details in Appendix A.
Initially, we illustrate the inversion capability of the proposed framework. Then, consider the text-guided image editing problem and demonstrate that our approach outperforms or comparable to significantly more expensive baselines.
Before diving into the main experiments, we present few generated samples using iCD-XL for steps in Figure 6. Additional quantitative and qualitative results are provided in Appendix B. The results confirm that the distilled model demonstrates strong text-to-image generation performance.
4.1 Inversion quality of iCD
Here, we analyze the reconstruction capabilities of iCD-SD1.5 under various configurations. Specifi- cally, we explore the contribution of the different pipeline components, such the number steps, preservation losses, and dynamic CFG, inversion performance. Our forward model is run without
7

---

Inversion
H
Configuration
LPIPS
DinoV2
PSNR 1
Unguided decuding setting
1DDIM
DDIM
0.167
0.834
22 22.98
Original
RCD CD2
0.332
0.632
17.75
B
[CD* CD*
0.317
0.649
18.42
ICD' CD'
0.276
0.715
19.19
E DED* + Cr CD*
0,484
0.554
16.
(CD* +
CD
0.218
20.01
G
ICD' + 2,
CD' 4
0.198
0.837
27
K
Guided decoding setting, w
IDDIM DDIM
0.179
0.531
11:12
1DDIM DDIM" d.CGG
0.279
0.726
19.08
H ICD' CD
0.476
0.550
13.87
I ICD* CD d.CGG
0.370
0.650
16.72
J. ICD' + G. CD + d.CG
0.317
0.698
17.98
K
ICD' + Cr CD + d.CGG
0.273
0.749
19.66
Table Exploration of ICD-SD1.5 configurations in terms of image inversion performance.
Figure 7: Influence the dynamic guidance and
preservation losses on image inversion with iCD.
CFG (w 1), while for the reverse model, we consider two settings: unguided (w 1) and guided (w = 8), both of which are important in practice.
Configuration To evaluate the inversion quality, we consider 5K images and the corresponding prompts from the MS-COCO dataset [56]. We measure the reconstruction quality using LPIPS [58], PSNR and cosine distance in the DinoV2 [59] feature space. As for the reference, the teacher inversion with a disabled CFG scale is considered. For the dynamic guidance, we use r = 0.7. The coefficients for the preservation losses equal to d 1.5 and X, 1.5.
Results The results are presented in Table 2. First, configurations (A-C) evaluate the number of the forward and inverse models inference steps. We observe that the reconstruction quality improves as the number of steps increases. our main experiments, we consider and 4 steps.
Then, (E-G) examine the preservation losses. In (E), we learn the forward model in the encoder [60] regime using the forward preservation loss only. This experiment reveals that the consistency loss contributes significantly to inversion performance. (F, G) show that both losses improve the inversion, with the latter approaching the quality of the teacher model.
Finally, we explore the dynamic CFG and preservation losses under the guided decoding setting (I-K) and compare them the setting (H), which does not employ any boosting techniques. From the configurations (I. J. K), we can see that all techniques provide sigmificant contribution to the reconstruction quality. In Figure 7, we visualize their influence on inversion. It can be seen that the dynamic CFG (I) is rather responsible for global object preservation, while the preservation losses (J,K) rather improve fine-gran fine-grained details. We note that the final configuration (K) provides comparable inversion quality to the unguided process while preserving the editing capabilities due to the activated guidance. More visual examples of inversion and quantitative results are in Appendix C.
4.2 Text-guided image editing
In this section, we apply the proposed iCD to the text-guided image editing problem. For the SD1.5 model, we use the Prompt-to- Prompt (P2P) method [61]. We vary two hyperparameters: the cross-attention and self-attention steps balancing between editing strength and preservation of the reference image. We also apply our approach to MasaCTRL [62] in Appendix D. For the SDXL model, we follow the ReNoise [25] evaluation setting and just change the source prompt during decoding according to [63].
Metrics We measure editing performance using both automatic metries and human-study. The former uses two metries: 1) to estimate the preservation of the reference image, we calculate the cosine distance between images the D2 feature space; 2) as an editing quality measure, we use the CL.PP score between the edited image and the target prompt. For human evaluation, we employ professional assessors who successfully completed assessment tasks. We show them the source and target prompts, reference image and two images produced with the methods under the comparison
8

---

Original
Ours. steps
NTI, 100 steps
NPI, 100 steps
InfEdii, 15 steps Edit-fr., 164 steps
wolf
blue
eyes
girl in dress
origami birds
storm trooper
Figure 8: Image editing examples produced by our method (iCD-SD1.5) and the baseline approaches.
COCO
PieBench
COCO
PieBench
0.255
0.27
Stccp
56
atpo
sore.
0.250
strpps
shps
sope
Ouas
NPL
NPL.
0.25
100
acgs
Ouus
NTL.
0.24
0.245
NPI 100 Haye
8 Salope
150
intibt 15 segge
8 steps
Clus
40
Edi-4.
164 steps
164 steps
Edit
0.4
DmnV22
preference,
Eluntnn penference,
Figure 9: Quantitative comparisons between different editing approaches based SD1.5: automatic metrics (left) and human preference study (right).
and ask the question: which of the edited images do you prefer more taking into account the editing strength and reference preservation?
Benchmarks In experiments, consider two benchmarks: PieBench [31] and manu- ally created COCO evaluation set based on text-paired images from the MS-COCO dataset [56]. PieBench [31] consists of various types of editing, for example, replacement, addition, deletion. We take 420 examples of realistic images, including all types of editing. COCO focuses solely on the replacement task as one of the most popular among practitioners. This benchmark contains 140 image-text pairs.
4.2.1 Text-guided image editing with iCD-SD1.5
Configuration To provide the editing with the SD1.5 model, we consider iCD using 4 forward and 4 reverse steps trained with both preservation losses (Ar = 1.5, X, = 1.5). In Appendix D, we also present the results for iCD using steps, which is not much worse than the step model. We set the hyperparameters of the dynamic CFG to 0.8 and maximum CFG scale to 19.0.

---

Original
RoNaie Turbo
ReNoise SDXLL. stops
ReNaiie LCM stops
Automatic
in the mouth squared caks
Configuratin
CLIP T
DinoV2
CLIP score,
C0CO benchmurk
0.472
0.748
ReNoise
0.705
RoNoise SDXI.
0.227
0.431
0.732
ReNoise LCM,
0.218
0.350
0.663
PieHtench
Ours,
0.254
0.707
0.858
RuNoise Turbo, 44
0.254
0.615
0.520
RaNoise SDXL, 150
0.234
0.642
0,850
pizza s0up
RoNoise 1 CM, 35
0.236
0.736
0.565
Human preferenee, %
COCO benchmark
PizBench
pink hoart
64 4.9
Ours, ReNoise Tubo,
1.1.1
RcNoise Tarbo, 44
4.9
5.1
RcNoise SDXL, 150
9: 2.8
RcNouse SDXL,
RoNoise LCM,
9G
5:t: 1.8
Ours, ReNoise L.M,, 35
24 4.2
coloed
photo
Table 3: Automatic metries (top) and human evalu-
ation (bottom) for iCD-XL and ReNoise [25].
Figure 10: Editing examples using XL models.
Baselines We compare our approach with four baseline approaches, which provide state-of-the the-art editing performance: Null-text Inversion (NTI), Negative-prompp Inversion (NPI) [32], InfEdit [26] and Edit-friendly DDPM [43]. All methods except Inffdit are diffusion-based approaches using more than 100 steps. Moreover, NTI employs the additional high-cost optimization procedure to improve inversion quality. InfEdit operates with the distilled diffusion model, latent consistency distillation [18], but utilizing virtual inversion. All methods use P2P and we vary all possible hyperparameter values to find the configurations that provide the best editing-preserrven
trade-off.
Results Figure 9 provides quantitative results for both benchmarks. We observe that the proposed iCD is comparable to the baseline approaches in most cases. Moreover, sometimes it can even outperform them while being multiple times faster. For instance, according to human preference on the COCO benchmark, our approach surpasses the InfEdit, NPI, NTI and Edit-friendly DDPM (50 steps). On the PieBench, it outperforms the InfEdit and Edit-friendly DDPM (50 steps). In addition, we provide qualitative results in Figure 8, which confirm the competitiveness of the proposed method. Additional visual examples can be found in Appendix D.
Notably, the performance of the proposed method is weaker on the PieBench benchmark compared COCO, according to both automatic and human-based metrics. We attribute this the increased complexity of editing tasks, which probably require more steps.
4.2.2 Text-guided image editing with iCD-XL
Configuration We consider the configuration using steps, T 0.7 and CFG scale equals 8.0.
Baselines We compare our approach to the recently proposed ReNoise [25], accurately following its guidelines. This method works with both distilled models (LCM-SDXL [18], SDXL-Turbo [19]) and original diffusion model, SDXL [1]. However, even for the distilled models, significant number of steps is required achieve decent performance.
Results The quantitative and qualitative comparisons are presented in Table 3 and Figure 10, respectively. According the human evaluation, iCD-XL outperforms all ReNoise configurations. Based on the automatic evaluation, our approach provides better reference preservation (DinoV2 and CLIP score, 1) while maintaining strong editing capabilities, as indicated by the CLIP score (T). We provide more visual examples in Appendix D.
Conclusion
The paper proposes generalized consistency distillation framework that enables both accurate image inversion and solid generation performance using a few inference steps. Accompanied by the recently proposed dynamic guidance, the distilled models demonstrate highly efficient and accurate image manipulations, making a significant step towards real-time text-driven image editing.
10

---

References
[1] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[2] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
[3] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-imate image diffusion models with deep language understanding. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolutinn image synthesis with latent diffusion models, 2021.
[5] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic rext-to-image synthesis, 2023.
[6] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024.
[7] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 18392-184005 2023.
[8] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks, arXiv preprint arXiv:2311.10000 2023.
[9] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yay Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[10] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.0 0169, 2023.
[11] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.
In International Conference on Learning Representations, 2022.
[12] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint 211.:223100.1889 2023.
[13] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[14] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In seveth Conference on Neural Information Processing Systems, 2023.
[15] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation, 2023.
[16) Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 03000, 2022.
11

---

[17] Liangchen Li and Jiajun He. Bidirectional consistency models. arXiv preprint arXiv:2403./8005, 2024.
[18) Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: universal stable-diffusion acceleration module. arXiv preprini arXiv:231 1.05556, 2023.
[19] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023.
[20] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[21] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality Liffusion-basse text-to-image generation. In International Conference on Learning Representations, 2024.
[22] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation, 2024.
[23] Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation, 2024.
[24] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024.
[25] Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. 2024.
[26] Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. 2024.
[27] Barak Meiri, Dvir Samuel, Nir Darshan, Gal Chechik, Shai Avidan, and Rami Ben-Ari. Fixed- point inversion for text-to-image diffusion models, 2023.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[29] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[30] Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 1.t0:22201..166,, 2020.
[31] Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting diffusion-basee editing with lines of code. International Conference on Learning Representa- tions (ICLR), 2024.
[32] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038-6047, 2023.
[33] Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative- prompt inversion: Fast image inversion for editing with text-guided diffusion models, 2024.
[34] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEF/VE Conference on Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
12

---

[35] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion-ben based style transfer with diffusion models. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 10146-10156, 2023.
[36] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[37] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. arXiv preprint 1xXiv:2211.124465 2022.
[38] Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing. arXiv preprint arXiv:2303. 15649, 2023.
[39] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: bexte real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6007-6017 2023.
[40] Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text- driven image editing using diffusion models. Proceedings of the IEEE/CVE International Conference Computer Vision, pages 7430-7440, 2023.
[41] Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint arXiv:2305.16807, 2023.
[42] Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu, Qilong Zhangli, et al. Improving gegative-prompe inversion via proximal guidance. arXiv preprint 2rXiv:2306.05414 2023.
[43] Inbar Huberman- Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint arXiv:2304.061400 2023.
[44] Manuel Brack, Felix Friedrich, Katharina Kormmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinaros Passos. Ledits++: Limitless image editing using text-to- image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[45] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through condition-annent sampling. In The Twelfth International Conference on Learning Representations, 2024.
[46] Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance limited interval improves sample and distribution quality in diffusion models, 2024.
[47] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[48] Tero Karras, Mïia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26571 2022.
[49] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02550, 2020.
[50] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787. 2022.
[51] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:220..12598, 2022.
13

---

[52] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing nigh-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[53] Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint wVXiv:2403.06807, 2024.
[54] Angela Castillo, Jonas Kohler, Juan C Pérez, Juan Pablo Pérez, Albert Pumarola, Bernard Ghanem, Pablo Arbeláez, and Ali Thabet. Adaptive guidance: Training-fii fareing---eeeeee acceleration of conditional diffusion models. arXiv preprint arXiv:2312. 12487, 2023.
[55] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
[56) Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[58] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR. 2018.
[59] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Ya- Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[60] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen Or. Encoding in style: A stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2287-2296, June 2021.
[61] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen -oh Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208.00 016666 2022.
[62] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 22560- 22570, 2023.
[63] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
14

---

A Training details of iCD
30
CD
0.5
ICD DDI
DDIM
Roomstrucion
0.3
F1D
24
22
20
0.1
1000
2000
3000
4000
5000
6000
2000
3000
6000
Training Iteration
Training iteration
(a)
(b)
Figure 11: Training dynamics of iCD-SDI.5 in terms of FID (a) and reconstruction loss (b).
First, following [20], we preliminary distill classifier free guidance using a conditional embedding added to the time step embedding. The goal is to use different and apply dynamic techniques during inference after consistency distillation. We perform CFG distillation for the following guidance scales: 1,8, 12, 16, 20 for SID1.5 and 1, 4,4, 6, 9, 10, 12, 13, 16, 18, 20 for SDXL.. At this stage, the model successfully approximates the guided teacher without hurting its performance.
Then, perform
consistency distillation using LoRA adapters with rank of 64 following [52]. We train forward and reverse adapters in parallel starting from the same teacher: CFG distilled SD1.5 or SDXL. For the SD1.5 model, we a global batch size of 512, and for the SDXL 128. All models converge relatively fast, requiring about 6K iterations with a learning rate 8e-6.
We find that the forward and reverse models provide promising generation and inversion quality for 3 or steps. The regularization coefficients for the forward and reverse preservation losses are A and 1155, respectively.
The iCD-SD1.5 models trained for 366h and the iCD)XL ones for ~68h 8 NVIDIA A100 GPUs. We present the training dynamics in terms of FID and reconstruction MSE for iCD-SD1.5 in Figure 11.
For SD1.5 distillation, we use a 2-0 subset LAION 2B, roughly filtered using CLIP score [64]. For SDXL, we collect ~7M images with resolution 2 1024, also curated to avoid poorly aligned text-image pairs and low quality images.
We set the following time steps for our configurations:
4 steps, 7 = 0.8: reverse model [259, 519, 779, 999]; forward model [19, 259, 519, 779]:
4 steps, 0.7: reverse model [259, 519, 699, 999]; forward model [19, 259, 519, 699];
3 steps 0.7: reverse model [339, 699, 999]; forward model [19, 139, 699);
B Image generation with iCD
We provide the generation performance of distilled model in Table 4. The dynamic CFG (r = 0.8, T = 0.7) degrades in terms of ImageReward, while improving FID due to the increased diversity [45, 46). The visual examples presented in Figures 16, 17.
C Image inversion
In Table 5, we provide comparisons between the 3- and 4-step configurations of iCD-SD1.5, which perform similarly. Figure 14 shows the image inversions provided by approach, NTI and NPI.
15

---

Configuration
FID
CLIP score
ImageReward
DDIM 50
18.73
0.266
0.201
CD
18.45
0.259
0.044
CD*
17.96
0.258
-0.009
CD d.CCG. == 0.8
16.25
0.254
-0.193
CD + d.C.G. 0.7
16.38
0.253
-0.217
CD* T = 0.7
17.66
0.251
-0.351
Table 4: Text-to-image performance of the SD1.5 model in terms of FID-5K, CLIP score and ImageReward for w. using prompts from the MS-COCO dataset.
Original
steps
steps
PieBench
0.200
Outs, steps
Ours, 3+33 steps
CLIP scope, +
0.255
mest
0.250
0.215
leopard
0.70
0.75
DinoV2.
(b)
Figure 12: Quantitative (a) and qualitative (b) editing results using 3- and 4-step iCD-SD1.5 configu- rations PieBench.
Original
Ouss, P2P
Ours, MasaCTRL
Ours, P2P
Ours, MasaCTRL
ramning corgi
jumping dog
stunding hour
rutsing his hand
Figure 13: Comparison between Prompt2Prompt and MasaCTRL applied our approach
Figure shows the image inversions compared to the ReNoise method. Table we present the time required to invert single image for different methods.
16

---

Configuration
I.IPS 4 DinoV2
PSNR
Unguided decoding setting
IDDIM DDIM
0. 167
0.834
22.98
E ICD*
CD + CD3 +L
0.198
0.837
22.27
E
ICD* + L
0.192
0.841
22.21
Guided decoding setting,
IDDIM
DDIM 50
0.479
0.534
14.12
IDDIM 30 DDIM 50 d.CFG
0.279
0.726
19.58
ICD* + &
CD L, + d.FGG
0.273
0.749
19.66
E
fCD*
L
CD L, d.F.G
0.263
0.770
19.76
Table 5: More image inversion results for 3- and s-sp iCD-SD1.... Dynamic CFG uses 0.7.
Method
Ours steps, SD1.5
NTL, SD1.5
NPI, SD1.5
Ours steps, SDXL
ReNoise, LCM-XL
Time, sees
0.959 .005
116.4 .1
9.95 + .03
1.56 = .07
6.75 + .52
Table The time required to invert a single image for different approaches.
D Text-guided image editing
Figure 12 compares two iCD configurations (3 and steps). We observe that both configurations perform similarly, with slight preference for the 4-step configuration. We present additional visual results on image editing using the iCD-SD1.5 model in Figure 18 and the iCD-XL model in Figure 19.
Figure 13 provides a few examples our approach combined with MasaCTRL [62] for non-rigid editing. The results demonstrate that method can be used with various editing methods.
We also present comparison with SDXL--Turbo using SDEdit [63] Figure (upper). We observe that SDXL Turbo significantly hurts reference image preservation due to stochasticity. This highlights the importance accurate image inversion for editing. Morcover, we compare our approach with Intruct-Pix2Pix [7], Figure 20 (bottom), and observe that outperforms Intruct-Pix2Pix in terms of both content preservation and editing strength while being training-free.
We present the annotation interface for the assessors in Figure 22. We calculate the confidence interval using bootstrap methods, splitting the human votes into 1, 000 subsets and then averaging the results and calculating the standard deviation.
E
Failure cases
We present some inversion failure cases in Figure 21. Our method sometimes oversaturates images for high guidance scales and struggles to reconstruct complex details like human faces and hands
F
Limitations
The iCD limitations include the requirement to predetermine boundary time steps before distillation, which may restrict the model flexibility. Additionally, the extra preservation losses necessitate increased computational resources during training. Furthermore, the editing method is susceptible to hyperparameter values, leading to inconsistent performance across different prompts. Finally, the quality of the current distillation techniques itself requires significant improvement, as does not consistently achieve high standards across various scenarios.
17

---

G
Broader impacts
Our work can significantly enhance tools for artists, designers, and content creators, allowing for more precise and efficient manipulation of images based on textual inputs. This can democratize high-quality digital art creation, making accessible those without extensive technical skills. On the other hand, the ability to edit images easily and realistically can be misused to create misleading information or fake images, which can particularly harmful and potentially influence public opinion and elections.
18

---

Original image
Ours, 8 steps
NPI, 100 steps
NTI. 100 steps
Figure 14: Inversion examples produced by the SDI.5-based models.
19

---

ReNoise Turbu 44 sleps
ReNoise LCC 35 steps
ReNoise SDXL steps
Original image
Ours, sleps
Figure 15: Inversion examples produced by the SDXL--based models.
20

---

Distillation d.CFG, v=0.8
Distillation d.CFG, r=0.7
Teacher
Distillation
A grey bird in the middle of a snow-covered ground
brown stuffed teddy bear sitting on top of couch.
An elephant at a zoo on a sunny day
Man and woman sitting next each other
panoramic view of cabin with stone fireplace.
A plate with steak, gravy, baked apples, and broccoli.
Figure 16: Generation examples using SD1.5, the proposed distilled method using 4 steps and dynamic CFG.
21

---

Distillation d.CFG, v=0.8
Distillation d.CFG, r=0.7
Teacher
Distillation
barred owl peeking out from dense tree branches
a dutch baroque painting of horse field flowers
bloody mary cocktail next to a napkin
cute illustration of horned owl with graduation cap and diploma
peaceful lakeside landscape with migrating herd of sauropods
A blue Ford F-150 coming around a curve mountain road
Figure 17: Generation examples using SDXL, the proposed distilled method using 4 steps and dynamic CFG.
22

---

Original
Ours, steps
NTI, steps
NPI, steps
InfEdiit, steps Edit-fr.. 164 steps
sculpture
lion
goat horse
sushi
cat
duck
starry sky
lion
Figure Additional editing examples produced by SD1.5-based models.
23

---

ReNoise, Turbo 44 steps
ReNoise, SDXL ReNoise, LCM
Original
Ours, 8 steps
150 steps
35 steps
zebra
watermelon
vegetables apples
>
mushrooms
bug
pie
yeti
rais
Figure 19: Additional editing examples using the SDXL-based models.
24

---

Original
Ours
TDXI-Tubbo
Ours
SDXI.TTrbo
colored photo
holding nuusorram
city skyline in buckgroinnd Ours
cal In water
Original
Instrvet-P2P
Iostruct-PP
womann warks through the foresr
dog 7OT
drgg rubobitt
lesparal color to the dog
Figure 20: Image editing results produced by our method and SDXL-turbo using SDEdit (upper) and Instruct-P2P (bottom).
Oripital
Inversio
Figure 21: Failure cases.
Figure 22: The human evaluation interface for the text-guided image editing problem.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions mentioned in the abstract and introduction sections carefully aligned with the methods and experimental sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: the limitations are discussed in the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper does not provide theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the technical details and include the code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
We attach the code in the supplementary material and will release it upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.cm co/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. "nips.cc/pppii cccppblic/guu wewgdueeeooeeeiieeeeeeeeeee wwteesummisiioPPooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
To the best of our judgment, we provide the details sufficient to reproduce the results. However, we do not discuss some low-level implementation details that are not critical for understanding, and refer the readers to our code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the confidence intervals for the main experiments and and explain their nature in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the configurations in each section of the experiments. Moreover, include additional information about the training in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpps://eett /etp:://nnuuiuueeeeeee wwt..cccpllliicccc ctw.e//teehiieeleeeeecc ines?
Answer: [Yes]
Justification: We have carefully read the NeurlPS Code of Ethics and conducted our research in accordance with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss potential positive societal impacts and negative societal impacts in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will provide guidelines for users upon acceptance of a paper control the use our models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We plan release code and models upon acceptance under the CC-by 4.0 license.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Human evaluation is performed by professional assessors. They are officially hired, paid competitive salary and informed about potential risks. We include the instruction in the supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Before involving participants in the evaluation, we provide them with detailed instructions addressing potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "N9bWHpZfvN", "reviewer": "Reviewer_H4g5", "summary": "This paper targets a novel problem of enabling image inversion and editing for models distilled with consistency distillation. \nThe authors identify the challenges of applying consistency distillation, which is designed for the denoising process, to the diffusion process and propose multi-boundary consistency distillation that achieves inversion with as few as 3-4 steps. A regularization term is proposed to guarantee the consistency between the forward and backward models. The authors further study dynamic CFG for performance improvements. The effectiveness is validated on SD 1.5 and SDXL models.", "questions": "The presented evaluations seem to focus primarily on 7 steps. While the performance is impressive compared to other methods, the higher performance reported by fDDIM50 makes me wonder if the proposed method can match the performance with more steps and what the marginal performance would be if we scale up the number of steps. \n\nMinor suggestion, doesn't affect rating:\nThe overall quality of this paper can be greatly improved by replacing Figure 1. The current one seems too simple and needs to be more informative. Figure 6 can even be a better alternative for Figure 1.", "limitations": "The authors have properly discussed the limitations in Appendix Section E.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper targets a very emerging and important problem of how to adapt the abilities DM models have to the accelerated version such as consistency-distilled models. \n\n- The paper is generally well-organized, and the presentation is easy to follow. \n\n- The empirical results are pretty strong given the very small number of steps used by the proposed method.", "weaknesses": "- While the overall problem is novel, and the solution demonstrates strong performance, almost every individual technique presented in this paper comes from another earlier work and makes the overall contribution less impactful.\n\n- The proposed framework is highly customized toward CD, and might not be able to generalize other DM model acceleration methods such as SD-turbo, DMD, and UFOGen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and valuable feedback! We respond to your questions below.\n\n1. *almost every technique comes from another earlier work...contribution less impactful.*\n\nOur techniques present a natural continuation of previous works without compromising novelty, in our opinion. Invertible Сonsistency Distillation and its corresponding training pipeline have not been formulated before, and dynamic guidance was not explored from the inversion perspective. We believe our approach makes an important contribution to the field, as it can be useful for future works on inversion-based problems.\n\n2. *The proposed framework is highly customized toward CD, not be able to generalize SD-turbo, DMD, and UFOGen.*\n\nUnlike GAN- or DMD-based distillation methods, CD exploits the ODE perspective of diffusion models. In our ablation study in Table 2E, we show that using ODE perspective via CD objective largely enhances our inversion method. The inversion of other distillation methods remains an open and interesting question. \n\n3. *Can the proposed method match the performance with more steps and what the marginal performance would be if we scale up the number of steps?* \n\nIn the table below, we present the results of the improved version of our model after more careful tuning. 8 steps of our inversion closely approaches the performance of the teacher inversion. Therefore, further scaling does not seem necessary. We will provide more details in the revision.\n\n| Metrics | Ours, 4+4  | DDIM, 50+50 |\n| -------- | -------- | -------- |\n| PSNR $\\uparrow$     | 22.81  | 23.07     |\n| LPIPS $\\downarrow$    | 0.179         | 0.167   |\n| DinoV2 $\\uparrow$    | 0.859         | 0.851     |\n\n4. *The overall quality of this paper can be greatly improved by replacing Figure 1*. \n\nThanks for the suggestion! We will update Figure 1 in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sl99pbDstu", "reviewer": "Reviewer_QtT7", "summary": "This work introduces invertible Consistency Distillation (iCD), which enhances text-to-image diffusion models by enabling effective encoding of real images into latent space. iCD achieves both high-quality image synthesis and accurate image inversion in just 3-4 inference steps.", "questions": "Please refer to the weaknesses.", "limitations": "This work has discussed limitations in Appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The adaptation of Consistency Distillation to tackle image editing tasks represents a valuable and promising research direction. \n2. The design of the forward and backward consistency distillation mechanisms is interesting. \n3. The experimental results are impressive, demonstrating that the proposed method achieves similar or even superior editing effects in fewer inference steps compared to existing models.", "weaknesses": "1. Discussion of Failure Cases: It would be beneficial to include a detailed discussion of failure cases. Understanding the scenarios where the method does not perform optimally can provide valuable insights and guide future research directions.\n\n2. More SDXL Demos Needed: The main paper (Figure 6) showcases demos using SD1.5, which, however, still suffers from generation issues like details of hands and faces. It is better to include more examples using SDXL (which offers improved performance) in the main paper. This would strengthen the evaluation and illustrate the capabilities of the proposed method more effectively.\n\n3. Comparison with SDXL Turbo (Adversarial Distillation): SDXL turbo is known for its excellent performance in one-step generation and can be used for editing with tools like SDedit. A comparative analysis of the editing effectiveness and efficiency between the proposed method and adversarial distillation methods would provide a clearer understanding of the advantages and potential trade-offs, offering a more comprehensive evaluation of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and valuable feedback! We respond to your questions below.\n\n1. *Discussion of Failure Cases*.\n\nWe present some inversion failure cases in the attached file (Figure 1). Our method sometimes oversaturates images for high guidance scales and struggles to reconstruct complex details like human faces and hands. We will add the discussion and more examples in the revision.\n\n2. *More SDXL Demos Needed*. \n\nThe paper has SDXL illustrations in figures 8 and 16. To strengthen the evaluation, we will provide more SDXL examples in the revision.\n\n3. *Comparison with SDXL Turbo using SDEdit*. \n\nThanks for the important suggestion! Please see the comparisons in the table below and the qualitative results in the attached file (Figure 2, Top). SDXL Turbo with SDEdit significantly hurts reference image preservation due to stochasticity, as confirmed by the DinoV2, image CLIP score and qualitative comparisons. This highlights the importance of accurate image inversion for editing. Moreover, our approach outperforms SDXL Turbo in terms of editing strength (ImageReward).\n\n\n| Method | ImageReward $\\uparrow$ | DinoV2 $\\uparrow$ | CLIP score, I $\\uparrow$ |\n| -------- | -------- | -------- |-------- |\n| Ours     | 0.473     | 0.726     | 0.873 |\n| SDXL-turbo     | 0.364     | 0.637     | 0.835|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qj0LVZt3LD", "reviewer": "Reviewer_Qhsn", "summary": "This paper extend the idea of consistency distillation to inversion for image editing. By training a separate consistency model where the consistency is enforced at noise space rather than latent space. Additional cycle consistency loss is employed for more accurate inversion.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an important problem: efficiency in image inversion. While there are lots of method improving the generation speed of diffusion models, the reverse direction of inversion is less tackled. This paper is an important contribution to this field. \n\n2. The shows promising results in fast image inversion and editing.", "weaknesses": "1. The description of the method is unclear. For example, section 3.1 and 3.2 are the main idea of the method, but it ignores an important component: the data. When training consistency distillation, we use perturbed real data, but since here we need to map to the noise, we need to have a coupling of the image and noise. How to obtain this coupling? Do we need to run inversion with the teacher model on a large dataset to obtain the training data? This is very important component of the method but I didn't find an explanation throughout the paper.\n\n2. The method basically needs two models, by employing CD in forward and backward directions, for fast inversion and generation. Although it is practically useful, I feel like it is a bit ad-hoc. The reason is that, if we enable two models, one for fast generation and the other for fast inversion, then any diffusion acceleration methods can be employed separately for two models. It would be more interesting to study this question: given a distilled diffusion model that can be sampled in few steps, can we use this model to do fast inversion?\n\n3. Related to above point. Two models introduce significantly more parameter overhead, and given that inversion based image editing is only only handle a small subset of image editing tasks, it becomes hard to justify whether it worths to double the parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and valuable feedback! We respond to your questions below.\n\n1. *The description of the method is unclear... we need to have a coupling of the image and noise.*\n\nAn interesting aspect is that neither additional data nor teacher inversion is required. Compared to the original CD, the only modification needed is the boundary condition. This is possible because the forward CD operates on the same ODE trajectories as the original CD (please see Figure 3 in the paper). The primary difference lies in the direction of the consistency function. Thus, image-noise couplings are \"implicitly\" created during training using one step of the ODE solver, much like in the original CD. We will provide more details in the revision to ensure clarity.\n\n2. *Given a distilled diffusion model, can we use this model to do fast inversion?*\n\nIt is an interesting question for future research. Since existing distilled models do not support reversibility like diffusion models, we believe that fast inversion of the already distilled models necessiates an extra model to learn image-noise connections. Our intuition is supported by extensive literature on GAN inversion, where an encoder is trained to perform inversion.\n\n3. *The method basically needs two models...I feel like it is a bit ad-hoc...Two models introduce significantly more parameter overhead....*\n\nWe believe that the distilled models need to be redesigned to handle bidirectional sampling (data to noise and noise to data): it can be either an extra input argument, LoRA adapters, or an additional model. As described in Appendix A, our method employs LoRA adapters as a highly convenient and compact way to enable invertibility. Specifically, we use a single diffusion model and simply activate the corresponding LoRA weights depending on which model is used (forward or reverse). Note that LoRA adapters take up a small portion of the total number of parameters (less than 10%). Thus, the parameter overhead is rather negligible compared to storing an extra model.\n\n4.  *inversion-based image editing only handles a small subset of image editing tasks*\n\nBesides image editing, fast inversion methods can be useful for other problems, such as anomaly detection [1], text-to-3D generation [2,3], image restoration [4].\n\n[1] Hend et al. Out-of-Distribution Detection with a Single Unconditional Diffusion Model\n\n[2] Lukoianov et al. Score Distillation via Reparametrized DDIM\n\n[3] Liang et al. LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching\n\n[4] Chihaoui et al. Blind Image Restoration via Fast Diffusion Inversion", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLwqyY7iKH", "reviewer": "Reviewer_hEvy", "summary": "The paper proposes distilation for both of forward and reverse path of ODE sampling of diffusion models in order to enable faster editing.", "questions": "Already written in the weakness part.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The overall framework is quite interesting, and applying the consistency distillation for forward and reverse process is novel.\nAlso the method of dynamic CFG is straightforward and seems to have meaningful inversion performance improvement even with CFG.\n\nThe proposed method can be combined to various image editing task and expected to show more efficient editing performance.", "weaknesses": "1. Although the method shows great performance in image editing with conditions, it feels that the editing method is limited on Prompt2prompt. Is it still possible to apply the model on non-rigid editing such as MasaCTRL? \n\n2. It seems that the most important contributions of this model is faster inversion & sampling. Please explicitly compare the inversion&sampling time with other baseline methods and put the table in main paper. \n\n3. In the experiment part, please compare the model with other training (or fine-tuning) based methods such as instruct-pix2pix or ControlNet. The current experiment only focuses on inversion methods, therefore further experiment would be more helpful for the manuscript.\n\n4. Does the performance degrade if the distillation step decreases? Please give more discussion and experimental results of the case of decreased steps such as single step of 4 steps.  \n\n5. minor comment, The Figure 1 (First figure) seems very redundant for me. I recommend a new figure which contains the brief summary of methodology such as combined version of Figure 3 and Figure 1. The figure in the current version make the paper seems unprofessional. As I like the overall idea, it would greatly improve the readability of the manuscript with changing the first figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and valuable feedback! We respond to your questions below.\n\n1. *it feels that the editing method is limited on Prompt2Prompt (P2P). Is it still possible to apply the model on non-rigid editing such as MasaCTRL?*\n\nOur model is not limited to P2P. Alternative editing methods, such as MasaCTRL, are also applicable in the same manner as to original diffusion models. The attached file (Figure 3) provides a few examples of our approach combined with MasaCTRL for non-rigid editing. The results show performance improvement due to more advanced editing methods.  We will add these results in the revision. \nPlease note that our work introduces a fast inversion method that is quite orthogonal to editing approaches. Therefore, we evaluate P2P as one of the most well-known approaches to compare various inversion methods fairly.\n\n2. *Compare the inversion&sampling time with other baselines*\n\nThanks for the valuable suggestion! We present the time required to invert a single image in the table below. We will add these results to the revision.\n\n\n| Method   | Ours 8 steps, SD1.5     | NTI, SD1.5     |      NPI, SD1.5   | Ours 8 steps, SDXL | ReNoise, LCM-XL \n| -------- | -------- | -------- | -------- | -------- |  -------- | \n| Time, secs     | 0.959+-.005     |116.4+-0.1  | 9.95+-.03      | 1.56+-.07 | 6.75+-.52\n\n\n3. *Compare the model with other training based methods*\n\nBelow, we compare our approach with Intruct-Pix2Pix and observe that it outperforms Intruct-Pix2Pix in terms of both content preservation and editing strength while being training-free. We also present a few visual examples in the attached file (Figure 2, Bottom). We will add this comparison to the revision.\n\n| Method | ImageReward $\\uparrow$ | DinoV2 $\\uparrow$ | CLIP score, I $\\uparrow$ |\n| -------- | -------- | -------- |-------- |\n| Ours, 8 steps    | 0.064     | 0.726     | 0.872 |\n| Instruct-P2P, 100 steps     | -0.227     | 0.708     | 0.850|\n\n\n4. *Does the performance degrade if the distillation step decreases?*\n\nThe scaling performance of our approach is similar to that of consistency distillation in text-to-image generation. Based on our experiments, the optimal number of steps is 6-8 (3-4 encoding + 3-4 decoding steps) and performance noticeably degrades for 2-4 steps (please see the tables below). We believe that future work on consistency models will greatly contribute to improving performance in fewer steps, which can then be transferred to our method.\n\nInversion:\n| Metrics | Ours, 4+4  | Ours, 2+2 | Ours, 1+1 \n| --------  | -------- | -------- | -------- |\n| PSNR $\\uparrow$     | 22.81     | 21.91 | 20.85\n| LPIPS $\\downarrow$    | 0.179          | 0.235 |  0.244 \n| DinoV2 $\\uparrow$    | 0.859             | 0.820 | 0.766\n\n\nEditing:\n| Method | ImageReward $\\uparrow$ | DinoV2 $\\uparrow$ | CLIP score, I $\\uparrow$ |\n| -------- | -------- | -------- |-------- |\n| Ours, 8 steps    | 0.064     | 0.726     | 0.872 |\n| Ours, 4 steps     |     -0.035 |   0.553  | 0.815|\n| Ours, 2 steps     | -0.428     | 0.498     | 0.761|\n\n\n5. *Figure 1 seems very redundant for me*\n\nThanks for the suggestion! We will update Figure 1 in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their constructive feedback, which will help us significantly improve our work. In our individual responses, we address the raised questions and concerns and we attach a PDF file with supporting qualitative results.

Meta-review: This paper extends distilled text-to-image diffusion models to fast editing by introducing invertible consistency distillation, which allows efficient image inversion and synthesis. All reviewers give positive scores after rebuttal. The problem is interesting, the method is novel, and the performance is significant. AC agrees this is a nice contribution on diffusion models and image editing, and thus recommends acceptance.
Decision: Accept (poster)