PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editing
Authors: Feng Tian
Yixuan Li
Yichao Yan
Shanyan Guan
Yanhao Ge
Xiaokang Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Measurements
Posterior Sampling
Image Editing
Inversion-free
Trianing-free.
Primary area: generative models
TL;DR: We present a sampling process based on posterior sampling for image editing.

Abstract: In the field of image editing, three core challenges persist: controllability, background preservation, and efficiency. Inversion-based methods rely on time-consuming optimization to preserve the features of the initial images, which results in low efficiency due to the requirement for extensive network inference. Conversely, inversion-free methods lack theoretical support for background similarity, as they circumvent the issue of maintaining initial features to achieve efficiency. As a consequence, none of these methods can achieve both high efficiency and background consistency. To tackle the challenges and the aforementioned disadvantages, we introduce PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt. Extensive experimental results indicate that the proposed PostEdit achieves state-of-the-art editing performance while accurately preserving unedited regions. Furthermore, the method is both inversion- and training-free, necessitating approximately 1.5 seconds and 18 GB of GPU memory to generate high-quality results.

Published as a conference paper at ICLR 2025
POSTEDIT: POSTERIOR SAMPLING FOR EFFICIENT ZERO-SHOT IMAGE EDITING
Feng Tian', Yixuan Li', Yichao Yan', Shanyan Guan", Yanhao Ge Xiaokang Yang 'MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2vivo Mobile Communication Co., Ltd ftf1021, lyx0208, yanylchac, es (guanshanyan, haloge]@vivo.com
ABSTRACT
In the field of image editing, three core challenges persist: controllability. back- ground preservation, and efficiency. Inversion-based methods rely time- consuming optimization to preserve the features of the initial images, which re- sults in low efficiency due the requirement for extensive network inference. Conversely, inversion-free methods lack theoretical support for background simi- larity, as they circumvent the issue maintaining initial features to achieve effi- ciency. As a consequence, none of these methods can achieve both high efficiency and background consistency. To tackle the challenges and the aforementioned dis- advantages, we introduce PostEdit, method that incorporates posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measure- ment term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt. Extensive experimental results indicate that the proposed PostEdit achieves state-of-the-art editing performance while accurately preserving unedited regions. Furthermore, the method is both inversion- and training-free, necessitating approximately 1.5 seconds and 18 GB of GPU memory to generate high-quality results. Code: https:: /ttp://gtttaaa thub.uu.b.... www.////T//////ooccccoccc
INTRODUCTION
Large text-to-image diffusion models Saharia et al. (2022); Pernias et al. (2024); Podell et al. (2024); Ramesh et al. (2022) have demonstrated significant capabilities in generating photorealistic images based on given textual prompts, facilitating both the creation and editing of real images. Current research Cao et al. (2023); Brack et (2024); Ju et (2024); Parmar al. (2023); Wu & la Torre (2022); Xu et al. (2024) highlights three main challenges in image editing: controllability, back- ground preservation, and efficiency. Specifically, the edited parts must align with the target prompt's concepts, while unedited regions should remain unchanged. Moreover, the editing process must be sufficiently efficient to support interactive applications. As illustrated in Fig. 1, there are two main- stream categories of image editing approaches, namely inversion- based and inversion-fit -fre methods.
Inversion- approaches Song et al. (2021a); Mokady et al. (2023); Wu la Torre (2022); Luberrann-Speeellge et al. (2024) first invert clean image to noisy latent (inversion phase) and then denoising the latent conditioned on the given target prompt to obtain the edited image (editing phase). However, directly inverting the diffusion sampling process inevitably introduces de- viations with the input image, due to error accumulated by the unconditional score term (discussed in classifier-free guidance (CFG) Ho & Salimans (2022) and proven in App. A.14). Consequently, the editing quality of inversion based methods is primarily constrained by the similarity in unedited regions. Several approaches address this issue by optimizing the text embedding Wu et al. (2023), employing iterative guidance Kim et al. (2022); Garibi et al. (2024), or directly modifying attention bias introduced by the unconditional term. However, the necessity of adding and subsequently removing noise pre- dicted by a network remains unavoidable, thereby significantly constraining their efficiency. Recent
layers Hertz et al. (2023); Mokady et al. (2023); Parmar et (2023) to mitigate the
corresponniig author:

---

Published as a conference paper at ICLR 2025
Target Conditions Embedding
Capy
Network
Inversion Based & Tuning Required
Iversion-Based
Xg
Inversion Free
Xole
Training Required
Inversion Free Training Free
Process of Iterative Generation
+h (0,1)
Noiey Measurement
X/
Posterior Distribution
p(xly) Optimization
Global Optimal Strategy
-E
Our Optimization Process Based Posterior Sampling
Figure 1: Different Image Editing Schemes. The inversion- irrersion-based method, illustrated in the top- left section, involves adding noise from pre-trained network to clean image. It then denoises the image based on a target prompt, though requires cone-connuming tuning to ensure background preservation. The top-right section discusses training-based inversion-free methods, which train a learnable model to achieve satisfactory results but have limited generalization capabilities. Our approach, outlined in the bottom section, is both inversion-free and training-free
methods Starodubcev et al. (2024); Li & He (2024); Kim et al. (2024) attempt to enhance the ac- curacy the iterative sampling process by training an invertible consistency trajectory, following the distillation process in the consistency models (CM) Song et al. (2023); Salimans & Ho (2022); Song & Dhariwal (2024); Luo et al. (2023b). Although this approach significantly reduces the ac- cumulation errors from the unconditional term, it cannot eliminate them. Moreover, the editing performance is sensitive to the hyperparameters (i.e., the fixed boundary timesteps of multi-step consistency models), and the training process generally demands hundreds of GPU hours.
Another category methods Brooks et al. (2023); Mou et al. (2024); Ye et al. (2023); Guo et al. (2024); Li et al. (2023); Wang et al. (2024) is inversion-free and thus significantly decreases the infer- ence time. The general idea is to train networks to learn to embed the given conditions into the noisy- to-image diffusion process. For example, ControlNet Zhang et al. (2023b) and T2I-Adapter Mou et al. (2024) train an extra network to encode the image-shaped conditions, e,g., depth maps, canny maps. However, these works highly rely on the accuracy of the input guidance structure, while most applications related to ControlNet involve customization. Some other works Zhang et al. (2023a), Zhang al. (2024b), Hui et al. (2024) employ a diffusion model trained on synthetic edited images, producing edited images in a supervised manner. This methodology obviates the need for inversion process during the sampling stage. Moreover, there a training-fres method satisfy inversion free requirement Xu et al. (2024). It adopts specific settings of the DDIM solver to leverage the advan- tages of CM to ensure the editing quality. Although these recent works can achieve fast sampling and accurate editing, the aforementioned problem remains unsolved since the diffusion sampling process Ho et al. (2020); Song et al. (2021a;b) is necessary. Therefore, all the inversion -fre methods cannot circumvent the accumulation errors caused by the unconditional score term in CFG.
In this work, we present an inversion- and training-free free method named PostEdit to optimize the ac- cumulated errors of the unconditional term in CFG based on the theory of posterior sampling Kawar et al. (2021; 2022); Chung et al. (2023); Zhang et al. (2024a;a); Lugmayr et al. (2022); Zhu et al. (2023); Song al. (2021b). To reconstruct and edit an image 2o, adopt a measurement term y which contains the features of the initial image, and supervise the editing process by the posterior log-likelihood density Va, log p(æ+ly)) With this term, we can estimate the target image through progressively sampling from the posterior p( p(zz|y) (b) referring to the Bayes rule. The above process is reasonable since the inverse problems of probabilistic generative models are ubiquitous in generating tasks, which are trained to learn scores to match gradients of noised data distribution (log density), and this process is also called score matching Song & Ermon (2020), Song & Ermon (2019), Karras et al. (2022) and Karras et al. (2024). y is defined according to the following inverse problem
y A(wo) n,
2

---

Published as a conference paper at ICLR 2025
where A is a forward measurement operator that can be linear nonlinear and n is independent noise. Hence, the posterior sampling strategy can be regarded as a diffusion solver and it can edit images while maintaining the regions that are required to remain unchanged with the measurement y. Also, instead of time-consuning training or optimization, our framework adopts an optimiza- tion process without requirements for across the network many times for inference, which can be lightweight taking about 1.5 seconds to operate and around 18 GB of GPU memory. Our contribu- tions and key takeaways shown follows:
To the best of our knowledge, theory of posterior sampling to text-guided image editing task.
the first to propose framework that extends the
We theoretically address the error accumulation problem by introducing posterior sam- pling, and designing an inversion-free and training-free strategy preserve initial features. Furthermore, we replace the step-wise sampling process with a highly efficient optimiza- tion procedure, thereby significantly accelerating the overall sampling process.
PostEdit ranks among the fastest zero-shot image editing methods, achieving execution times of less than 2 seconds. Additionally, the state-of-the-e CLIP similarity scores on the PIE benchmark attest the high editing quality of our method.
2 PRELIMINARIES
2.1
SCORE-BASEE DIFFUSION MODELS
We follow the continuous diffusion trajectory Song et al. (2021b) to sample the estimated initial image an. Specifically, the forward diffusion process can be modeled as the solution to an Itô SDE:
dx = f dt + g.dw.
(2)
where f is defined as the drift function and g denotes the coefficient noise term. Furthermore, the corresponding reverse form of Eq. 2 can be written as
da
f (x) V (e)) di godhū
where tö represents the standard Brownian motion. As shown in Song et al. (2021b), there exists corresponding deterministic process whose trajectories share the same marginal probability densities as the SDE according to Eq. 2. This deterministic process satisfies an ODE
1 ot of logp((z)) dt.
dx f((x)
(4)
The ODE can compute the exact likelihood of any input data by leveraging the connection neural ODEs Chen et al. (2018). To approximate the log density of noised data distribution Va log (x) each sampling step, network sa (2 () is trained to learn the corresponding log density
Peo.erppaaall || 1) - Va, log log
(5)
2.2 DDIM SOLVER AND CONSISTENCY MODELS
The DDIM solver is widely applied in training large text-to-image diffusion models. The iterative scheme for sampling the previous step is defined as follows
/1 occo( t)
ft-1 = Van-1
11-00-160001 (),
Vay
where Eg (2x t) is the predicted noise from the network. According to Eq. 6, the sampling process can be regarded as first estimating a clean image æa, and then using the forward process of the diffusion models with noise predicted by the network to the previous step æ;-1- Therefore, the predicted original sample & defined as
Lg = It
VI an (I
(7)
ar
3

---

Published as a conference paper at ICLR 2025
Latent consistency models Luo et al. (2023a) apply the DDIM solver Song et al. (2021a) a) to predict Eo and use the self-consistency of an ODE trajectory Song et al. (2023) to distill steps. Then the Do is calculated by the function fo c, t) through large timestep, where is defined in Eq.
0z,0(2,
fo(z, Cakip (1)z Cout (1)
(8)
at
is denoted encoded in the latent space. The loss function of self-consistency is defined as LCD (0,0 V Ez,,,n (dl (So (Zim
bn+1) C, fo (*", e, c,,)))
where S an estimation of the evolution of the 2t,, from h441 using ODE solver V.
2.3 POSTERIOR SAMPLING IN DIFFUSION MODELS
After obtaining sg( t), we can infer an unknown æ Rd through the degraded measurement R" Specifically, in the forward process, it is well-posed since the mapping æ + TRd - IR" many-to-one, while ill-posed for the reverse process since one-to-many when sampling the posterior p(zolyy, where it can not be formulated as functional relationship. To deal with this problem, the Bayes rule is applied to the log density terms and we can derive that
Væ, log p(z, 1y) Vv. logp(z;) + Va, pgy(())
(10)
where the first term in the right side hand of the equation is the pre-trained diffusion model and the second one is intractable. The measurement can be regarded as a vital term that contains the information of the prior p(æ), which supervises the generation process towards the input images.
In order to work out the explicit expression of the second term, existing method DPS Chung et al. (2023) presents the following approximation
p (x))= (z0
p(y) ân = E Re [2o]
(11)
where the Bayes optimal posterior 2o can be obtained from given pre-trained diffusion models or Tweedie's approach iterative descent gradient for the case of VP-SDE or DDPM sampling. Hence, each step can be written as p( according to Eq. 10.
When the transition kernel is defined, since the solvers utilize the unconditional scores estimate Zp. the measurement term is then introduced through gradient descent way to optimize
1 f (0 <) + V/ |yy A(mo)l1 e N (0,
(12)
where the function ʃ is defined as the approximation of the unconditional counterpart of (/1,11/2,,3 3) and 77 denotes the learning rate.
METHOD
We propose a novel sampling process integrated with tailored optimization procedure that incor- porates the measurements and Langevin dynamics to enhance the quality of image reconstruction and editing. The adopted SDE/ODE solver is based on DDIM, as described in Eq. Denote E(ra), E Rº where is encoder and æa is initial image. Our method operates in latent space and leverages the theory of the posterior sampling to correct the bias from the initial features and introduce the target concepts. The core insight is using y in the form of Gaussian dis- tribution, estimated žn and Langevin dynamics as the optimization terms to correct the errors of the sampling process. The importance of reconstruction and the algorithm are introduced specifically in (Sec. 3) The implementation details of the editing process are illustrated in detail (Sec. 3.2). PostEdit takes around 1.5 seconds and 18 GB memory costs on single NVIDIA A100 GPU.
3.1 POSTERIOR SAMPLING FOR IMAGE RECONSTRUCTION
The quality of reconstruction is crucial indicator for evaluating the editing capabilities of method. To preserve the features of the background (areas unaffected by the target prompt), Mokady et al. (2023) introduces technique for fine-tuning the text embedding to mitigate errors caused by the
4

---

Published as a conference paper at ICLR 2025
7-1
Add noise
Zy
Esitimate 2g
Use to optimize of
4 Return %
Process
1|+6
aull
Generation
20
Iterations
Figure 2: Method Overview. The latent representation of initial image Do is zo. It is adding noise randomly to 2T and then En is estimated from æy through diffusion ODE solvers. After that, there are two optimization terms relating to ε., the given measurement y and random noise term e, which is applied to optimize calculated εn while avoids solutions falling in local optimality. Then the optimized &o adding noise to timestep T according to the noise scheduler. This process operates recursively and finished till द is converged to zo. where = is the finally optimized output.
null text term, demonstrated in App. A.14. However, this approach is time-consuming, and there a pressing need to enhance editing performance. To address this challenge, we propose a method that enables fast and accurate reconstruction and editing process.
Specifically, there are four steps in our method: (I) We add noise to 2u following the DDPM noise schedule until zy (N, I). Unlike the iterative inversion process used in Mokady et al. (2023) (DDIM inversion), where multiple network inferences are required, the added noise here is directly sampled from (0, I). As result, this process adds random noise directly to the clean image in a single step, significantly reducing the computational time. (2) Existing SDE/ODE solvers, such as DDIM and LCM, are employed to estimate 2g. (3) To ensure that &o aligns more consistently with the background and target prompt features of the original image, it is optimized using two L2 norm terms related to the defined measurement y and εn respectively. Additionally, Langevin dynamics is employed to avoid convergence to local optima. (4) Finally, by progressively applying the above process to update the mean and variance of the Gaussian distribution in the Step (1) according to predefined time schedule, we obtain zū with consistent initial features and accurate target characteristics respectively as converges to The presented algorithm corresponding to the above process is shown in detail in Fig. 2 for image reconstruction or editing and Alg. A.15 for image reconstruction. The measurement y. defined in Eq. 17, can be simplified as a masked observation. For deeper understanding, we recommend referring to Kawar et al. (2021).
Our method requires a large text-to-image diffusion model as input and we select Stable Diffusion (SD) Rombach et al. (2022). Given that SD is trained on the dataset containing billions of images, the generated result has strong randomness relating to the same prompt. Therefore, if we directly apply posterior sampling strategy shown in Eq. 12 to acquire 2o from ZT ~ NN I) by leveraging SD to inference noise, En differs greatly from the ground truth Zu- Moreover, reconstructing an image with given text prompt starting from N (0, I) usually gets poor results for SD due to the bias caused by the unconditional term in CFG. Conversely, posterior sampling has good reconstruction performance when leverages the diffusion model trained on small datasets, for example, FFHQ Karras et al. (2019) and ImageNet Deng et al. (2009) as shown in Chung et al. (2023); Zhang et al. (2024a). We experimentally discover that the gap between these two kinds of models is the inconsistent layouts of each estimated image εg. while the features of the target prompts generated ε.. Specifically, the layouts of & generated by the scores inferred by the networks trained on FFHQ and Laion-5B Schuhmann et al. (2022) for intermediate timesteps shown in App. A.4. Therefore, due to the editing and reconstruction trade-off issue, it is much more challenging for high-quality image editing and reconstruction by leveraging large text-to-image models.
successfully introduced into the
5

---

Published as a conference paper at ICLR 2025
To address the editing and reconstruction trade-off issue, we present weighted process that intro- duces the features of initial data into the estimated &y as shown in the following proposition.
Proposition 1. The weighted relationship between the estimated &o and the initial image 2in to correct evaluated 2o is defined (0 w 0.1)
2ú
w) 20 w Zin.
(13)
where w is a constant govern the intensity of the injected features.
By additionally introducing Zin (weighted by w) during sampling, we can produce more high- fidelity and similar layout with the input image, which is critical for applying posterior sampling for image reconstruction and editing.
Remark 1. Eq. 13 is reasonable since (1-w) and w- w are regarded as constant. Hence, this process does not essentially influence the sampling process of distribution 2t-1 which is shown specifically in the Proposition 2.
E (W (z8,, of I)),
In order to adapt Eq. 10 to the DDIM solver shown in Eq. 6 adopted by SD, derive it as
Z zo log p(z)) Σ, v) V logp(zolzz) Z logp(y) zc)
(14)
calculate the scores towards Zg straightly inspired by Chung al. (2023) and Zhang et al. (2024a). The measurement settings for image reconstruction are listed in the App. A.2
Proposition 2. Suppose ze is sampled from time marginal distribution of (zily), then
2 ~ Egg (z0. 1)-
(15)
satisfies the time marginal distribution conditioned on (Zt-1 11y), where zi is obtained Eq. 13. (Proof shown in Appendix 16)
Propsition 2 ensures that -,-1 sampled from the Gaussian distribution (with mean = and variance oi-1) still satisfies the constraint of the posterior sampling Eq. 10. Therefore, we can present the following scheme optimize the estimated &o and run Langevin dynamics Welling & Tch (2011):
||A
yll
(1) (k+1) = (1 w) Zu + w Zin h- --VV 20
120
201 20
+
+ 2he. (16)
2m²
Here, A (-) is identical to (-) as defined in Eq. 17 and h is the step size. or and are hyper- parameters detailed in Appendix A.2. Additionally, e ~ I). Fig. 15 and Fig. 16 in Ap- pendix A.10 demonstrate that PostEdit can achieve high-quality reconstruction outcomes without requiring any tuning process. Eq. 16 is reasonable since the two terms that multiplied by the step size have the same descent direction towards to the ground truth zg- Additionally, Langevin dy- namics is employed to search for solutions that achieve global optimum. Since the Step (1) shown Fig. 2 is different from the process of DDIM inversion from the initial image to noise, which involves adding noise Eg (z,,t, Gin)) inferred by the network at cach step Mokady et al. (2023) (where Cizi represents the prompt describing the content of the initial image), PostEdit is much more efficient mentioned before. Considering that no information related to the initial image is incorporated into the noised distribution, the term involving the measurement y, as defined in Eq. 16, introduced correct errors in the initial features caused by the unconditional term in CFG. This ensures background consistency. All parameter settings are detailed in Appendix A.2. The detailed process of image reconstruction outlined in Alg. 2 of Appendix A.15.
3.2 POSTERIOR SAMPLING FOR IMAGE EDITING
In this section, we present details of the posterior sampling process for high-quality image editing using the DDIM solver Luo et al. (2023a), outlined in Eq. 6. Unlike the ODE solver used in the image reconstruction task described Sec. 3.1, image editing requires the solver with higher accuracy to estimate e. The measurement for image reconstruction and editing is defined as
N (Pz, 02,021)
(17)

---

Published as a conference paper at ICLR 2025
Algorithm Posterior Sampling for Image Editing
1: Require:
Diffusion model éø, step size h, posterior sampling steps N. diffusion solver steps n, image æg, measurement y, weight ur target prompt Ctgt coefficients of diffusion sampler Cskip and Cout, encoder &, decoder D, noise schedule (t()), (t),, posterior sampler sequence (ri);=0 and diffusion sampler sequence {4)}==d
2: ( in = 20
3: for N 1 do
4: for j - to do
5: Sample 2j N (a (tz) 20, o² (tt)t
20 = Caii (1)z; + Cour (t) (2 Ctgtt))
6:
7: end for
8:
= 20
9:
for k to T T do
10:
Sample E ~ (0,
11:
(k+1) (1 - w) zu 13: Sample 2T,-1 N(2 (T)
20
w Zin - h 2000
2n
20
+
|LA
2mz
u11
2he
12:
end for
,7 I).
14: 20 = Caki (t)= + Cout (t) (# (e - Cane
15: end for
16: 20 = D (zo)
Return 20
where P E 10 represents a masking matrix composed of elementary unit vectors. This measurement setup not only serves specialized configuration for image editing but also demon- strates its capacity to deliver high-quality image reconstruction results, even when 20 is masked. The settings in Eq. 17 have been validated to yield high-quality reconstructions, as shown in Sec. 4.3, demonstrating the method's effectiveness in preserving the features of the initial image.
Furthermore, to minimize the number of sampling steps, improving the accuracy of the estimated 20 is crucial. According to the experimental results presented in LCM, the superior denoising capabil- ities of the LCM solver Luo et al. (2023a) are demonstrated surpass those of the DDIM solver in both speed and accuracy. Consequently, we utilize the LCM solver, distilled from models based on the DDIM solver, to markedly improve the convergence rate and produce more accurate &o estimates that closely align with the target prompt in fewer than four steps. The measurement characteristics y, defined in Eq. 17, involve randomly masking each element of zo with given probability. Since one of the optimization terms focuses only a small portion of the initial image, both terms Eq. 16 guide the gradient descent in the same direction. As the sampling process progresses, the edited Zegt gradually inherits features from both 2g and the target prompt by selectively replacing the necessary attributes. The experimental results of different settings for the optimization deñned in Eq. 16 are presented in Sec. 4.4. The rest of the process mirrors the reconstruction phase, allow- ing us to progressively achieve the edited Øg. In summary, the algorithm's procedure is detailed in Alg. I, with implementation specifics provided in Appendix A.2.
4
EXPERIMENTS
4.1 EXPERIMENT SETUP
To ensure fair comparison, all experiments were conducted the PIE-Bench dataset Ju et al. (2024) using the same parameter settings specified in Appendix A.2 and a single A100 GPU to evaluate both image quality and inference efficiency. The PIE-Bench dataset comprises 700 images with 10 types of editing, where each image paired with a source prompt and a target prompt. In our experiments, the resolution of all test images was set to 512 x 512. For the reconstruction experiments, we set the initial and target prompts to be identical across all test runs. Additional settings, including forward operators, provided in the Appendix A.2.
7

---

Published as a conference paper at ICLR 2025
Background Preservation
Efficiency
Method
PSNR
LPIPS 10
MSE 10
SSIM X.10
Timet
NTI
25.58
7.98
4.37
77.02
~: 120s
NPI
9.11
76.14
iCD
19.64
17.13
13.50
66.48
1.85
DDCM
18.00
17.74
18.94
64.01
Ours
24.39
9.00
4.75
72.74
1.5s
Table 1:
Quantitative Comparisons of Image Reconstruction.
All of the comparison methods
include strategies specifically designed for image reconstruction.
Background Preservation
CLIP Similurity
Efficiency
Method
PSNR
LPIPS
MSE
SSIM
Whole
Edited
Time
x100
NTI
27.50
5.67
3.40
85.03
25.08
21.36
1220s
NPI
25.81
7.48
4.34
83,44
25.52
22.24
15s
PnP
22.31
11.29
8,31
79.61
25.92
22.65
240s
DI
27.28
5.38
3.25
85.34
25.71
22.17
~60
iCD
22.80
10.30
7.96
79.44
25.61
22.33
1,888
DDCM
28.08
5.61
7.06
85.26
26.07
22.09
~-2
TurboEdit
22.44
10.36
9.51
80.15
26,29
23.05
~12s
SPD
28.86
3.42
2.33
86.86
25.54
21.50
~30
GR
25.03
7.29
4.71
83.34
25.83
22.43
~300
IP2P
19.65
17.99
26.26
75.19
24.93
21.71
~00
OmniGen
19.63
15.06
38.76
72.29
25.18
21.77
7000s
ScodX
18.79
17.52
20.82
74.93
25.76
22.34
~7s
Ours
27.04
6.38
3.24
82,20
26.76
24.14
1.55
Table 2: Quantitative Comparisons of Image Editing, indicates models that benefit from SDXL-Turbo's improved inference.
4.2 QUANTITATIVE COMPARISON
Image Reconstruction. The methods have special design for image reconstruction are compared: NTI Mokady al. (2023), NPI Miyake et al. (2023), iCD Starodubcev al. (2024) and DDCM Xu et al. (2024). The results of quantitative comparison are shown in Tab. 1. Although NTI and NPI achieve better performance on the listed metrics, their computational time costs are substantially higher, exceeding ours by at least an order of magnitude. Compared to the highly efficient inversion- free method, DDCM, PostEdit demonstrates significantly superior performance. Image Editing. We compare our method against recent inversion-bart editing approaches: NTI, NPI, PnP Tumanyan et al. (2023), DI Ju et al. (2024), iCD, DDCM, TurboEdit Deutch et al. (2024), SPD Li et al. (2024) and GR Titov et al. (2024), IP2P Brooks et al. (2023), OmniGen Xiao et (2024) and SeedX Ge al. (2024)). The comparison evaluated from three aspects: background consistency, CLIP Radford et al. (2021) similarity, and efficiency. The experimental results shown in Tab. reflect that PostEdit achieves SOTA performance on editing, which are the "Whole" and "Edited" metrics of the CLIP similarity and the results significantly better than others. For efficiency, our model is highly efficient with a runtime less than seconds. It is worth noting that runtime is slightly higher than TurboEdit Deutch et al. (2024), which is mainly due to different baselines. Specifically, TurboEdit employs SDXL-Turbe Sauer et al. (2023) while our framework is based LCM-SDi.5 Luo al. (2023a). As shown in Appendix A.11, SDXL-Turbo Sauer et al. (2023) is almost 2.5 times faster than LCM-SD1.5 Luo et al. (2023a). We believe the efficiency our framework can be further improved we adopt a more efficient baseline like SDXL-Turbo In terms of background preservation, our method achieves the best MSE result among all methods with a runtime of less than seconds. The following section presents additional qualitative results, further illustrating the superiority of our framework in editing capabilities and background preservation, while maintaining high efficiency.
and training basedd image
4.3 QUALITATIVE COMPARISON OF RECONSTRUCTION AND EDITING
Image Reconstruction. We present results of qualitative comparison in Fig. 3. The experiments indicate that PostBdit demonstrates greater robustness and high quality generation ability compared to NTI Mokady et al. (2023), NPI Miyake et al. (2023) and iCD Starodubcev et al. (2024). Specifi- cally, we compared the reconstruction quality across four distinct categories of images: single-object
See Appendix A.12 to get instructions for TP2P, OmniGen, and ScedX based the input and target prompt.
8

---

Published as a conference paper at ICLR 2025
Input
NTI
NPI
DDCM
iCD
Ours
Figure 3: Qualitative Comparison of Reconstruction. It takes 1.5 seconds for method to reconstruct the input image, and the time is 1.8s, 2s, 15s, and 120s for iCD, DDCM, NPI, and NTI, respectively. Our framework can faithfully reconstruct the foreground object and the background.
images, complex backgrounds, multi-object scenes, and cartoon images. The inversion-free method DDCM Xu et al. (2024) fails to faithfully reconstruct the input images, supporting our claim made in Sec. 1. While other methods yield better results in the given cases, they require significantly longer processing times to achieve competitive outcomes. Therefore, our approach offers the best overall performance when considering inference efficiency, stability in generation, and image quality. More reconstruction results complex objects are shown in Fig. 15 and Fig. 16.
Image Editing. The qualitative comparisons of the image editing results are shown in Fig. 4. The effects of text insertion, deletion, and substitution are provided. PostEdit effectively highlights the features present in the target prompt, which aligns with the quantitative results shown in Tab. 2. To present these findings more clearly, we selected the best- best-performing classical methods, and their results are shown in Tab. 2. For comparison of the other baseline methods, please refer to Fig. 12 in Appendix A.7. Additionally, the visualized experiments demonstrate that our method successfully preserves the original features. More comparison results can be found in Appendix A.7.
4.4 ABLATION STUDY
In this section, we conduct various ablation studies and present the results to demonstrate the effec- tiveness of our framework. (a) We remove the optimization component shown in Eq. 16 and directly apply the adopted SDE/ODE solver to estimate æg. The experimental results indicate that the edited images lack background preservation. For instance, the slanted bicycle example shown in the first row of Fig. 5, the staircase on the left side of the original image is transformed into car in the edited image. (b) We modify the masked probability of our measurement y. Notably, there is discernible difference between the edited images and the input images. (c) We investigate the influence Proposition on the experimental outcomes, which highlights the effectiveness of the parameter concerning background similarities. Additionally, the quantitative results, detailed in Tab. 5, highlight the adopted configurations of PostEdit achieve optimal generation performance.
5 CONCLUSION AND LIMITATION
In this work, we address the errors caused by the unconditional term in Classifier-FF Freffiee--Feeeeee Guidance by introducing the theory of posterior sampling enhance reconstruction quality for image editing.

---

Published as a conference paper at ICLR 2025
Input
OmniGen
SPD
NTI
PnP
DDCM
TurboEdit
Ours
squirre: sitiigg top of wooden alatform reading
a suep of coffee with drewing of testip Lion. puttzd on the woodew. table
R Wovese stonw.traoperr with olue hair werring a sklrt
o beettorrfty fand flowers
Figure 4: Qualitative Comparison of Editing. Our method performs better than the others in aligning with target prompts while maintaining the background similarity.
Input
(a) w/o. Optim. (b) w/o. Mask
(c) w/o. Znn
(d) Full
slantedd rusty mountRin bioycle ow road front building
dog sitting ow wooden oheir
geeseng clol wow.n. is holding dog Figure 5: Ablation Studies. We show the results without the optimization process shown in Eq. 16, the measurement y defined in Eq. 17 and shown in Proposition 1.
By minimizing the need for repeated network inference, our method demonstrates fast and accurate performance while effectively preserving background similarity, as evidenced by the results. UI- timately, our approach tackles three key challenges associated with image editing and showcases state-of-the-ars performance in terms of editing capabilities and inference speed.
Limitation: PostEdit faces challenges in representing highly specific scenes. For example, describ- ing "a man raising his hand" is considerably more difficult compared to the structured input formats used in ControlNet-related methods. Furthermore, its ability to maintain background consistency is limited and requires improvement. Additionally, the quality and speed of generation are strongly influenced by the performance of the underlying baseline models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by NSFC (62201342), and Shanghai Municipal Science and Tech- nology Major Project (201SSHZDZX01022, Authors would like appreciate the Student Innova- tion Center of SJTU for providing GPUs.
REFERENCES
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. Domain-agnostic tuning-encoder for fast personalization of text-to-image models. In SIGGRAPH Asia, 2023.
Omri Avrahami, Kfir Aberman, Ohad Fried, and Daniel Cohen. Break-a-scene: Extracting multiple concepts from single image. In SIGGRAPH Asia, 2023.
Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinario Passos. Ledits++: Limitless image editing using text-to-image models. In CVPR, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. CVPR, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W. Cohen. Subject-driven text-to-image generation via apprenticeship learning. In NeurlPS, 2023.
Hansam Cho, Jonghyun Lee, Scoung Bum Kim, Tae-Hyun Oh, and Yonghyun Jeong. Noise map guidance: Inversion with spatial context for real image editing. In ICLR, 2024.
Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. ICLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models. CoRR, 2024.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components esti- mation. In ICLR Workshop, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text-driven image editing using diffusion models. In ICCV, 2023.
Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning, CoRR, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. CoRR, 2024.
11

---

Published as a conference paper at ICLR 2025
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. SEED-X: multimodal models with unified multi uulti-granullrity comprehension and generation. CoRR, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. In ICLR, 2024.
Inhwa Han, Serin Yang, Taesung Kwon, and Jong Chul Ye. Highly personalized text embedding for image manipulation by stable diffusion. CoRR, 2023.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, Di Liu, Qilong Zhangli, Jindong Jiang, Zhaoyang Xia, Akash Srivastava, and Dimitris N. Metaxas. Proxedit: Improving tuning-free real image editing with proximal guidance. In WACV, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prome prommpt image editing with cross-attention control. In ICLR, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and Ying Shan. Smartedit: Exploring complex instruction-ben based image editing with multimodal large language models. In CVPR, 2024.
Inbar Huberman-Spieee Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. CVPR, 2024.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and
Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing. CoRR, 2024.
Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexci A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pup inversion: Boosting diffusion-based editing with 3 lines of code. In ICLR, 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine, Analyz- ing and improving the training dynamics of diffusion models. In CVPR, 2024.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: solving noisy inverse problems
stochastically. In NewrlPS, pp. 21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurlPS 2022, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
Dongjun Kim, Chich-Hsie Lai, Wei- Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ODE trajectory of diffusion. ICLR, 2024.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2025
Sharon Lee, Yunzhi Zhang, Shangzhe Wu, and Jiajun Wu. Language-informed informed visual concept learn- ing. In ICLR, 2024.
Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. In NeurfPS, 2023.
Liangchen Li and Jiajun He. Bidirectional consistency models. CoRK, 2024.
Ruibin Li, Ruihuang Li. Song Guo, and Lei Zhang. Source prompt disentangled inversion for boosting image editability with diffusion models. In ECCV, 2024.
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference. CoRR, 2023a.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: universal approach for transferring knowledge from pre-trained diffusion models. In NewrIPS, 2023b.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. CoRR, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusson models. In CVPR, 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang. Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In SIGGRAPH, 2023.
Pablo Pernias, Dominie Rampas, Mats L. Richter, Christopher Pal, and Marc Aubreville. An effi- cient architecture for large-scale text-to-image diffusion models. In ICLR, 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Pengchong Qiao, Lei Shang, Chang Liu, Baigui Sun, Xiangyang Ji, and Jie Chen. Facechain-sude: Building derived class to inherit category attributes for one-shot subject-driven generation. In CVPR, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. CoRR, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image image diffusion models for subject-driven generation. In CVPR, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
13

---

Published as a conference paper at ICLR 2025
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. CoRR, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jit- sev. LAION-SB: an open large-scale dataset for training next generation image-text models, In NeurIPS, 2022.
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent Y. F. Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing, In CVPR, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In JCML, 2021a.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In ICLR, 2024.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In NeurlPS 2020, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
Nikita Starodubcev, Mikhail Khoroshikh, Artem Babenko, and Dmitry Baranchuk. Invertible sistency distillation for text-guided image editing in around steps. CoRR, 2024.
Vadim Titov, Madina Khalmatova, Alexandra Ivanova, Dmitry P. Vetrov, and Aibek Alanov. Guide- and-rescale: Self-guidance mechanism for effective tuning-free real image editing. In ECCV, 2024.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. CVPR, 2023.
Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan. Uni- tune: Text-driven image editing by fine tuning diffusion model on a single image. TOG, 2023.
Bram Wallace, Akash Gokul, and Nikhil Naik. EDICT: exact diffusion inversion via coupled trans- formations. In CVPR, pp. 22532-22541, 2023.
Qixun Wang, Xu Bai, Haofan Wang. Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. CoRR, 2024.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Ya- sumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Sorieut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In CVPR, 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: encod- ing visual concepts into textual embeddings for customized text-to-image generation. In ICCV, 2023.
Max Welling and Yee Whye Tch. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
14

---

Published as a conference paper at ICLR 2025
Chen Henry Wu and Fernando De la Torr. Unifying diffusion models' latent space, with applica- tions to cyclediffusion and guidance. CoRR, 2022.
Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In CVPR, 2023.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. CoRR, 2024.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape
guided object inpainting with diffusion model. In CVPR, 2023.
Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with language-guided diffusion models. In CVPR, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, 2023.
Bingliang Zhang, Wenda Chu, Julius Bernrr, Chenlin Meng, Anima Anandkumar, and Yang Song. Improving diffusion inverse problem solving with decoupled noise annealing. CoRR, 2024a.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. In NeurlPS, 2023a.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control text-to-image diffuuson models. In ICCV, 2023b.
Shiwen Zhang, Shuai Xiao, and Weilin Huang. Forgedit: and forgetting. CoRR, 2023c.
Text guided image editing via learning
Shu Zhang, Xinyi Yang. Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, and Ran Xu. Hive: Harnessing human feedback for instructional visual editing. In CVPR, 2024b.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. Inversion-based style transfer with diffusion models. In CVPR, pp. 10146-10156, 2023d.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In CVPR, 2023.
A APPENDIX
A.1 RELATED WORK IN IMAGE EDITING
Previous works in image editing can be broadly categorized into two paradigms: inversion-based and training-base based.
Inversion-based Methods. Several works Zhang et al. (2023d); Ruiz et al. (2023); Gal et al. (2023) focus on modifying the training process of diffusion models to incorporate the informa- tion from the initial images. Specifically, the optimization process can be operated two different spaces: textual space and model space. In textual space, set of methodologies Dong et al. (2022); Valevski al. (2023); Han et al. (2023) aim to optimize textual embeddings to perform various edit- ing tasks effectively. In model space, research line Ruiz et al. (2023); Qiao et al. (2024); Avrahami et al. (2023) intends to further updates modules from the base model to enhance reconstruction ca- pabilities. For example, several studies Kawar et al. (2023); Shi et al. (2024); Zhang et al. (2023c) optimize both textual embeddings and model parameters to ensure content consistency following non-rigid editing or localized distortions. For forward-based inversion. it can also be divided into two categories, which DDIM inversion and DDPM inversion. Previous works like Mokady et al. (2023) and Miyake et al. (2023) designed to approximate the inversion trajectory to deal with
15

---

Published as a conference paper at ICLR 2025
accumulated errors. Dong et al. (2023) focuses optimizing the text embedding, which is then in- terpolated with the target embedding during the editing process. Some approaches Han et al. (2024); Cho et al. (2024) aim to bypass the time-consuming optimization processes of the aforementioned methods while preserving their reconstruction capabilities. Inspired by normalizing flow models Dinh et al. (2015; 2017) 7), EDICT Wallace et al. (2023) reformulates DDIM processes by simultane- ously tracking two associated noisy variables at each step during inversion. These variables can be exactly derived from one another during the sampling phase.
Training-base Methods. Since some advanced approaches in zero-shot or few-shot settings quire time-consuming optimization Ruiz et al. (2023); Gal ct al. (2023); Mokady ct al. (2023) or highly sensitive to hyperparameters Cao et al. (2023); Kawar et al. (2023); Hertz et al. (2023); Sutberman-Spiegelle al. (2024), several studies Brooks et al. (2023); Ye et al. (2023) aim to train task-specific models with substantial amounts of data directly transform source images into target images under user guidance. Instruction-based editing Brooks et al. (2023); Zhang et al. (2023a); intuitive approach for image manipulation, allowing users input
Xie al. (2023) provides command-style text instead of providing an exhaustive description. For image inpainting, a group of methods Huang et al. (2024); Wang et al. (2023) focuses on completing missing parts of an image under text guidance. Additionally, image translation Isola et al. (2017); Zhang et al. (2023b) seeks to transform the source image into target domain, such as converting night to daytime or sketch to natural image, Another type of training-based method is content-free editing, which Ruiz et al. (2023); Wei et al. (2023) aims to preserve the high-level semantics of the source images in the final results. Content-free editing can be further categorized into subject-driven customization and suribute-driven customization. Subject-driven customization Wei et al. (2023); ei al. (2023); Arar et al. (2023); Chen et al. (2023) is designed to capture the identity of the target and generate novel images that place it in new contexts. In contrast, attribute-driven customization Lee et al. (2024) focuses extracting and manipulating attributes in more fine-grained manner.
A.2 IMPLEMENTATION DETAILS
The main hyper-parameterr of the PostEdit are briefly summarized in Tab. 3.
SD Model. We adopt LCM-SD1.5 for all the experiments Luo et al. (2023a). Parameters of Consistency models. Cskip and Cout shown in line 6 of Alg. 1 are set to 0 and 1 for most cases respectively.
in Alg. 1. N is set to 5 for schedule (T To ensure higher efficiency and quality at the same time, nN is sampled through
NN N (Vāzza, VI vz)ā
(18)
where / is set to 501 generally following the DDPM scheduler Ho et al. (2020). Additionally, n is set to for the sequence of diffusion sampler { to further improve inference speed. The parameter T' shown in line 16 of Alg. is set to 100 to ensure optimal quality. For Eq. 16, m set as 0.01 for both the reconstruction and editing task while corresponds to the timestep of the DDPM scheduler. Generally, we apply Eq. for step to estimate 2a, and then according to the following schedule to make zτ, progressively converge to 2p.
{T: (501, 401, 301, 201, 101, 1).
(19)
The parameter w is usually set to minimal value such as 1. for most cases or 0 and 0.2 for easy and hard Additionlll,,, h is initially set to 1c-5 for image editing and reconstruction tasks. It is dynamically adjusted at each recursion step, as described in lines 9 to 12 of Alg. 1, using the following equation:
h h= (1 11 k
-=11++(000011) - (0.01-1)) h.
(20)
where l: and T are the same with definition of line 9 of Alg.
ODE Solvers. We adopt the solver of LCM Luo et al. (2023a) distilled from Dreamshaper v7 fine-tune of Stable-Diffusion v1-5 for images editing task. For reconstruction, different solvers, for instance, DDIM Song et al. (2021a), DDPM Ho et al. (2020), and Song et al. (2023) based on Stable Diffusion Rombach et al. (2022) are able to work out satisfied reconstruction quality.
16

---

Published as a conference paper at ICLR 2025
Notation
Values
Description
0.5
Appearance probability of Matrix shown Eq. 17
Optimization Steps
100
Operating steps Eq. The weighting coefficient of Proposition.
0.1
Table 3: Main Hyper-parameters.
Background Preservation
CLIP Similarity
Efficiency
Method
PSNR
LPIPS
MSE
SSIM
Whole
Edited
Time
f= 0.3
27.20
6.09
2.91
82.77
25.93
22.40
~v.55
f= 0.7
24.43
12.16
6.06
77.64
26.73
24.28
~ 11,5s
Optimization Steps 50
25.49
9.39
4.85
79.95
26.61
23.59
~ ~1.1s
Optimization Steps 150
26.59
7.19
3.77
82.05
26.51
23.47
~1.8s
0.3
27.00
8.50
4.39
82.70
26.34
23.49
11.5s
0.5
27.75
5.47
2.84
83.61
25.83
22.19
~1.5s
24.01
10.92
6.24
77.28
26.45
23.45
11,55
Ours Defiuult
27.04
6.38
3.24
82.20
26.76
24.14
1.S5
Table 4: Quantitative Results of Hyperparameter Sensitivity Analysis
Probability of Masked Features. We use the probability equal to 0.5 for randomly mask pro- cess, which represents whether one of the latent features is masked not.
Measurement y Used for Better Quality of Image Reconstruction. chosen from the following Eq. 21 to further improve the reconstruction ability of PostEdit, which are defined as linear and nonlinear operations relating to initial image 20 in latent space
The measurement y can be
y N (IFP o
(21)
where F and P denote the 2D discrete Fourier transform matrix and the oversampling matrix with ratio k/n respectively for Eq. 21. However, the forward operator term shown in Eq. 21 reflects poor editing ability, and all our editing and reconstruction results all based on the measurement shown in Eq. 17.
Oversampling Matrix. We set a shown in Eq. 21 to 0.01 and use an oversampling factor k 2
and 8.
2D Discrete Fourier Transform Matrix.
The Fourier transform is defined as
M-1 Σ
F [u, r) VMN
Σ
I [x, u) exp
-j2m
y
M
N
(22)
-=0
0, 1,..,, M 1;
where fxx y) denoted as two-dimensional discrete signal with dimension M N obtained by sampling at superior intervals in the spatial domain. x and are discrete real variables and discrete frequency variables, respectively. according to Eq. 22.
this paper, the z0 represented as matrix and operated
FFHQ Model. We adopt the ffhq- flmq-100mpp with a size of MMB as the baseline model for all the experiments relating to the FFHQ dataset.
A.3 HYPERPARAMETER SENSITIVITY ANALYSIS
The hyperparameters used for PostEdit are listed in Tab.3, and its performance under different set- tings is presented in Tab.4. We conduct the following hyperparameters sensitivity analysis:
Appearance Probability of in Matrix P. A higher probability (e.g., 0.7) improves CLIP Similarity but degrades Background Preservation metrics (e.g., PSNR and SSIM). Conversely, a lower probability (e.g., 0.3) favors background preservation at the expense of CLIP similarity.
Optimization Steps. Reducing the number of steps, such as 50, decreases computation time but negatively impacts performance across most metries. Increasing the steps to 150 offers slight performance improvement but reduces efficiency. The chosen configuration of 100 steps strikes a balance between quality and runtime.
17

---

Published as a conference paper at ICLR 2025
Weighting Coefficient. Setting to results in poor performance for both background preser- vation and editing capabilities. While increasing w enhances background consistency, editing performance remains suboptimal.
Our Configuration. The default settings strike a balance across all metrics, achieving competitive results background preservation, editing alignment, and efficiency.
A.4 COMPARISON THE IMAGES LAYOUT OF DIFFERENT DATASETS
In this section, we present comparison of the layouts of the estimated Do at different intermediate timesteps, inferred by the diffusion models trained on the SD and FFHQ datasets respectively.
In Fig. 6, we present three independent sets of results for both SD and FFHQ. each containing nine different instances of & selected from outputs various iterations. The first three rows display the results for SD, while the remaining rows correspond to FFHQ. Each set is tasked with generating the same target image based on the same initial image. From left right, the level of noise progressively decreases.
Notably, the layouts for SD are more varied, with inconsistencies in the cat's appearance, its position relative to the mirror, and the mirror's appearance across the three images. This contrasts sharply with the results from FFHQ, where the layouts consistently feature a centered face surrounded by stable background.
To verify that this property is consistently observed in results based on the FFHQ model, we present additional examples in Fig. 7. As move from bottom to top, the noise gradually decreases, while from left to right, there are 10 different examples. Each image represents the estimated 2 from different iterations.
A.5 QUANTITATIVE RESULTS OF ABLATION STUDY
To better reflect the effectiveness of our adopted settings, we also conduct a quantitative results of the ablation study shown in Tab. 5. The results further verify the performance of all settings of PostEdit.
Background Preservation
CLIP Similarity
Efficiency
Method
PSNR
LPIPS x102
MSE 103
SSIM x 102
Whole"
Edited
Time
No Posterior Sampling
21.31
16.88
73.21
26.38
23.28
~s
No mask
28.31
4.61
2.64
84.15
25.17
20.94
~ 1.55
No Zin
24.01
10.92
6.24
77.28
26.45
23.45
1..55
Ours Full
27.04
6.38
3.24
82,20
26.76
24.14
1.55
Table 5: Quantitative Comparisons of Ablation Study.
A.6 RESULTS OF LONG-TEXT EDITING
We conducted long-text editing experiments, with the results presented in Fig. 8, Fig. 9, Fig. 10 and Fig. 11. These results demonstrate that the editing capabilities of PostEdit extend beyond simple word replacements.
A.7 MORE EDITING RESULTS
Here, we qualitatively compare with other baselines including NPI, GR, DI, and IP2P, SeedX. The results shown in Fig. 12. The results support the conclusion in the manuscript.
A.8 RECONSTRUCTION RESULTS OF DIFFERENT FORWARD OPERATORS
Images in Fig. 14 are reconstructed through different forward operators as shown in Eq. 17 and Eq. 21. The corresponding quantitative comparison of image reconstruction of different measure- ments is shown in Tab. 6. The corresponding quantitative comparison of image reconstruction of different measurements shown in Tab. 6.
18

---

Published as a conference paper at ICLR 2025
Background Preservation
Efficiency
Mcasurement
PSNR
LPIPS X10#
MSE 1gs
SSIM 100
Time+
Eq. 21
24.90
7.60
4.31
74.03 72.74
1.58
Eq. 17 (Used)
24.39
9.00
4.75
1.58
Table 6: Quantitative Comparisons of Image Reconstruction using different Measurements.
A.9
INTERMEDIATE RESULTS
The intermediate state for different iterative steps are shown detailed in Fig. 13.
A.
ADDITIONAL RECONSTRUCTION RESULTS
The additional reconstruction results are exhibited in Fig. 15 and Fig. 16. Additionally, We compare reconstruction quality of different methods show in Fig. 19. The results reflect the effectiveness of PostEdit to reconstruct high frequency information.
A.11 COMPARISON BETWEEN LCM-SD1.3 AND SUXL-TURBO
Fig. 17 illustrates the inference speed of LCM-SD1.5, which is utilized in method, alongside SDXL-Turbo. The results indicate that TurboEdit Deutch et al. (2024) may not be faster than our method, despite its reliance the advanced baseline model, SDXL-Turbo. All experiments were conducted on a single NVIDIA A100 GPU with 80GB of memory.
A.12 GENERATING EDIT INSTRUCTION FOR COMPARING WITH INSTRUCTION- BASED IMAGE EDITING APPROACHES
We provide GPT4-o with the following instruction shown in Fig. 18 to convert the differences be- tween the input prompt and the edited prompt into editing instructions suitable for IP2P and SeedX.
A.13 USER STUDY
We invited 34 anonymous volunteers to rank the preferred results of image editing results. The results are evaluated by the quality of background preservation and features aligned with the given target prompt. The feedback is shown in Tab. 7 and Tab. 8 and the preference represent a vote of the participants. The results indicate that PostEdit outperforms the compared baselines and is the most popular approach for both image reconstruction and editing tasks.
Method
Ours
iCD
DI
SPD
NTI
PnP
DDCM
Preference (Editing)
81
22
8
|
23
5
12
Method
TurboFdit
NPI
GR
IP2P
SeedX
OmniGen
Preference (Editing)
24
18
30
Table 7: User Study of Image Editing.
Method
Ours
NTI
NPI
DDCM
iCD
Preference (Reconstruction)
101
66
56
11
Table User Study of Image Reconstruction.
19

---

Published as a conference paper at ICLR 2025
Algorithm 2 Posterior Sampling for Image Reconstruction
Require: Diffusion model cø, diffusion sampler 20 (+), posterior sampling steps N. step size h, image Zo, measurement y, weight w, initial prompt Cini, encoder &, decoder 'D noise schedule a(I), (t), optimization steps N and posterior sampler sequence (7i) )
Zrx ~ (0, f).
for N to do
Zo = 20 (Zr,, Cini)
for j = N2 do
c ~ (0, I).
(++1) (1 w) 20 20
(5)
+ Zin - h- V 22(0)
|
zoll
+
11 v
) + 2he.
20
2m
end for Sample n- ~ N (2)ar)) " - 1).
end for DD (z0) Return 25p
A.14 CLASSIFIER FREE DIFFUSION GUIDANCE
According to CFG Ho & Salimans (2022), the generation process is governed by the conditional score, which can be derived as follows
(xx) plc xx)
Va, log pag =V. log
p(c)
=V., logp(æx; + p(c xx.)
(23)
Va, logp(c)
Va, log log)(2;; + Va, logp(c xx).
And then the term Va, logp(c xx) can derived as
xx, logg (c xx) = Vx, log (xt c) - Vx, logp (xt)
(24)
(Eo (x,,t, Eo (xt, t)).
0z
Substituting the above term into the gradients of classifier guidance, we can obtain
Ea (Xr, L, c) = Es (Xe, t c) - /1 V, logp (c xt) Eg (x,, 1, c) + w (eg (x,,,,)) Eg (x,,t) (w + 1)ea (xt, () weg (xe:)).
(25)
Clearly, there is an unconditional term (also known as the null-text term) that directly contributes to the bias in the estimation of 2y when the DDIM inversion process is applied under Classifier-Free Guidance (CFG) conditions. To mitigate this influence. tuning process is typically required to optimize the null-text term, ensuring high-quality reconstruction. Furthermore, to achieve better alignment between the generated image and the text prompt, as well to enhance image quality, is often necessary utilize larger value of w. However, this can exacerbate cumulative errors, leading to significant deviations in the acquired latent representation.
A.15 ALGORITHM FOR IMAGE RECONSTRUCTION
The overall process of image reconstruction by applying posterior sampling is shown specifically in Alg. 2.
The differences between the reconstruction and editing tasks is the ODE solvers applied estimate h. and the initial and target prompts remain the same with each other for image reconstruction. All the parameter Settings are shown specifically in App. A.2.
20

---

Published as a conference paper at ICLR 2025
A.16 PROOF OF PROPOSITION 2
According to Eq. 14, the distribution of - depends on 2n and 2o. The marginal distribution
relating timestep ( - 1 can be rewritten by Proof. We first factorize the measurement conditioned time-marginal p (Z12 yy) by
( ly,c)= = /f p (2:-11 ne y) dz) dz.
(26)
// p (zz (, ze. cc 20 24, y,, c) day dze,
according to the proposition 1, the above equation can be written as
p (z y.c) = /f p p(z czc) p. 2 2, y, c) dzy dz,
11 p(z y, c) (((( w) zo + w Zin)
Zz, y, c)
p(a--1 ((1 (1-w) Zo + w Zin)
Zeo y/ c) ((1 -w) 20 + w zin) dz
@
p ((z Iy) [(1 ([-w)) -w) p (z0 y, + w (Zin Y'l
p (z (1 (1-w) (w Zu 2, 4) dl ((1 - w) 2o + w Zin) dz:
@
p (Zz 1y) (11-((1)) w) 20 y,
p (2 ((1-) Zo Z, y) d ((1 w) zo) dz
m
((z 1z) Zz 2r, c) (2:-1
Zo, Zz y) dzo dzt
lf p(z yy) (zo) Zz, y, c) (z:-1
20) dzo dz.
Epe ,.. (z:-1 zo)
(10)
Be (2- Zo, -i-1
(27)
where (i) is dues to independent relationships and (ii) is derived by variable substitution and is the given target prompt. (iii) is derived directly according to the process defined in Eq. 15, whose independent variant is substituted by zo instead of =1f.
21

---

Published as a conference paper at ICLR 2025
Figure 6: Layouts of evaluated outputs for the same objects at different intermediate timesteps.
Figure Layouts of evaluated output for various objects and timesteps.
22

---

Published as a conference paper at ICLR 2025
Original (Left);
edge of paved road modern building beige walls. The bike features a black and red fiame, with thick rreaded tires designed for rugged lerain. The handlebars are straight ergonomic grips, the front suspension with conerete curbs, metallie miilings, and muted when tones adding to the setting's atmosphore.. The lighting is natural, softly highlightnng the bicycle's glossy finish and sturdy construction.
sleck mountain bicycle parked slight stant, positioned
indicates it's huilt for off-mad performance. sursunding industrial,
Edited (Right):
completely rusty mountain bicycle parked slight slant, posilionce the edge of a paved road moder building with beige walls. The ontire frame, including the handlebars, stspension fork, and wheel rims, is heavily corroded, covered in thick layer
overtaken any remmants paint or glass. The chain and gear system are seized crusted rust, rendering them nop- funciioaaal The tires are cracked and worn, with nast even creeping onto the spokes and hubs, The industrial setting, with concrete curbs and metallic railings, provides stack contrast to the biks'' decayed abandoned appearance. The natural lighting accentustes the uneven texture and reddish-broww huc of the rust, making the dicycle look it has been left to the elements for many years.
Figure 8: Example of Long-text Editing.
Original (Left::
A beautifully crafted round coko with a rustic charm sits elogantly on a wooden plate made from natural tree slice, showcasing its earthy nesthetic. The cake adomed with light orange frosting. finely textured with crumb casting wraps around its sides, giving a warm and inviting appearance. The top of the cake features smooth swirl of white cream, adding touch of contrast and sophistication to the design. Surrounding the cake green leaves, small bow] filled with vibrant erangs crumbs, and muted mug with wooden spoen resting neurby. The setting is completed with dark fabric background and woven busket, creating homey and natural ambisnce that complements the rustic presentation of the cake.
cozy seene, featuring elements like white coramic vase with fresh
Edited (Right):
heantifully crnfted square cake with a mustic charm sits elegantly wooden plate made from naturn tree slice, showeasing its earthy aesthetie. The cake adomed with light orange frosting, finely textured with crumb costing that wraps around sides, giving il
and invitin uppearance. Tbe contrast and sophistication to the design despite its square shape Surnenndig the cake is a eazy scene, featuring elements like a white ceramic with fresh green leaves, small bowl filled with vibrant orange crumbs, and muted green mag with weoden spoon resting nearby. The setting is completed with a dark fabric background and woven basket, creating a homey and natural arbiancc that complements rustic presentation
the cake features smooth swirt of white cream, adding touch of
square
Figure 9: Example of Long-text Editing.
23

---

Published as a conference paper at ICLR 2025
Original (Lefty:
matching saucer, placed polished wooden table. latte art on the surface features an intricate leaf-like design made with stenmed milk, showcasing the skill of the barista. The warm lones of coflee complereent rich wood grain the lable, creating a cozy and inviling stmosphere.
while cerumii cup
beautifully enalled
Edited (Right):
matching white smeer,, placed polisted wooden table. The latte art on the surfice has been transfommed into the detailed outline of lion's head, cratted with precisson using steumed milk. mome facial features depicted smooth intricate swirls. showessing impressive artistic skill. warm tones of the coffee remain unchanged, blending harmonionsly with the rich wood grain of the table, preserving the cozy and inviting atmosshere.
white ecramic cup filled with beautifully crafted
Figure 10: Example of Long-text Editing.
Original (Leffy:
vibraut bird with bright yellow fetthers and an orange-sed chest perches gracefully on thin tree branch lush. forest. The bird's 's sharp black head contrasis beautifully with its colortia plumage. The background is filled with soll-focus green lesves and the textured bark of trees, creating serene and natural setting.
Edited (Right):
fully crocheted bind, crafted entirely from colorful yarn, sits delicately on thin tree branch lush, green forest. The bird's crocheted black with visible loops stitches, giving distinetly handmade iрpearnnce. The wings, tnil, and legs intricately shaped with consistent crochet patters, making the look cutirely crafted frem yam. The artificial, textured crochct design stands out against the natural elemeats, with the tree branch and the vibrant green leaves amocsphree while the filly crocheted bird adds whimsical and creative touch.
body made soft, bright yellow yam, with onnge-red chest,
background remaining unchanged. The setting msintains screne, natural
Figure 11: Example of Long-text Editing.
24

---

Published as a conference paper at ICLR 2025
Input
NPI
GR
IP2P
SeedX
DI
Ours
a squirrel sitting on top of wooden, platform reading booke.
cup of coffee with drawing of tutipp Lion putted the wooden table
PA Spuuly
A Wemeste storu-trooper with blue hair wearing a shirt
A &bbutterfltg and flowers
Figure 12: Additional Comparison Results of The Remained Baselines.
25

---

Published as a conference paper at ICLR 2025
Step
Step 2
Step 3
Step 4
Step 5
POITE
Figure 13: Intermediate Results.
26

---

Published as a conference paper at ICLR 2025
Input
Eq.(17)
Eq.(11)
Input
(q.(17)
PREEviG Speli
Eq.(21)
palll
Input
Eq.(17)
(q.(21)
Figure 14: Additional Comparison Results Based on Different Measurements.
27

---

Published as a conference paper at ICLR 2025
Input
Reconstruction
Input
Reconstruction
Input
Reconstruction
Input
Reconstruction
Figure 15: Additional reconstruction Results.
28

---

Published as a conference paper at ICLR 2025
Input
Reconstruction
Input
Reconstruction
Input
Reconstruction
Input
Reconstruction
Figure 16: Additional reconstruction Results.
29

---

Published as a conference paper at ICLR 2025
SDXL-Turbo
0.8s
LCM-SD1.5
-2.s
0.4s
0.8s
1.2s
1.6s
2.0s
Figure 17: Comparison of Inference Speed.
You expert in image editing tasks, responsible for generating clear and concise editing instructions based on two image descriptions.
Given the following descriptions:
Original image description: "(original_prompt"
2. Edited image description: "[edited_promptty prompt)"
Please infer the differences between these descriptions and provide single concise instruction to describe the necessary edits to the original image. The instruction should be as brief as possible. For example:
the difference color change, use expressions like "Make the bicycle rusty."
it involves adding elements, use expressions like "Add a red hat to the person."
it involves removing content, use expressions like "Remove the text from the image."
Output format:
json
Copy code
"instruction": "Your instruction content"
Figure 18: Instruction Generation process GPT.
30

---

Published as a conference paper at ICLR 2025
Input/ / PSNR
ICD / 23.36 dB
NPI / 24.84 dB
DDCM / 17.75 DB
GR/ /RR/Z 21.82 dB
NTI / 25.27 dB (~120ss)
SPD / 25.68 dB (~30s)
PostEdit' 26.15 dB (-15s)
Input/
NPI/
21.21 211/2
/ 17.55
GR GR/10 19.05 dB
NTI / 23.54dB (-120s)
SPD / 23.80 dB (~30s)
PostEdit' 24.45dB (-15s)
Figure 19: Comparison of Different methods for Reconstruction of High Frequency Details. represents 10000 optimization steps are adopted for this result.
31


Reviews:
[{"review_id": "9eeEF84DpB", "reviewer": "Reviewer_6fQq", "summary": "This paper tackles the issues caused by the unconditional term in Classifier-Free Guidance (CFG) by integrating the theory of posterior sampling to improve reconstruction quality for image editing. By reducing the reliance on repeated network inference, the proposed method, PostEdit, achieves fast and accurate performance while successfully maintaining background consistency. The results seem promising.", "questions": "Please see the Weaknesses part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. By extending the theory of posterior sampling to text-guided image editing tasks, the proposed method, PostEdit, eliminates the need for both inversion and training. \n\n2. PostEdit is one of the fastest zero-shot image editing approaches, achieving execution times of under 2 seconds on an A100 GPU.", "weaknesses": "While the paper presents an interesting approach, some details and experimental results are not sufficiently comprehensive. Key aspects of the methodology are not fully elaborated, and additional experiments would be beneficial to further validate the claims. \n\n1. The subfigure \"Our Posterior Sampling Process\" in Fig. 1 is difficult to understand. It is unclear what exactly it is meant to represent. Additionally, how does it highlight the advantages of the proposed algorithm?\n\n2. The authors mention that the proposed algorithm runs in less than 2 seconds, but it is not clear how many denoising steps are actually performed. In Algorithm 1, the values of N and n are not specified. These details are crucial because if the number of denoising steps is high, it could naturally lead to slower execution. It would also be beneficial to visualize some intermediate results for better clarity.\n\n3. Regarding the editing process, how is the mask generated and provided? There is a lack of details on this aspect.\n\n4. The current editing experiments only involve replacing a few words in the prompt, which seems quite limited. Does the method support more extensive editing operations? More explanation and additional experiments would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5sPmhvMT0", "reviewer": "Reviewer_Psjc", "summary": "This paper introduces a diffusion-based method designed for image reconstruction and editing, which operates within an encoded latent space. The proposed Langevin update rule for the latent vector at time $t+1$ consists of three components: a convex combination of the initial clean latent vector and the latent vector at time $t$, a negative gradient sum that guides the vector toward the initial noisy latent vector, and the measurement, and added Gaussian noise. \n\nThe authors evaluate the proposed method through image reconstruction and editing experiments, providing qualitative and quantitative comparisons with state-of-the-art approaches. In editing experiments, the method outperforms the competitors in CLIP Similarity and achieves competitive results in other metrics while also being one of the fastest methods.", "questions": "- In equation 16, the gradient symbol is missing before the large brackets. The same issue occurs in line 13 of Algorithm 1 and row 710 in Algorithm 2.\n\n- For a more comprehensive background, it may be beneficial to include a citation to [1], which introduced a method for solving linear image restoration problems in cases with noisy observations.\n\n[1] Kawar, Bahjat, Gregory Vaksman, and Michael Elad. \"SNIPS: Solving noisy inverse problems stochastically.\" Advances in Neural Information Processing Systems 34 (2021): 21757-21769.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed algorithm is both inversion-free and training-free, contributing to its fast performance. The manuscript is well-written and clear. In editing experiments, the algorithm outperforms state-of-the-art approaches in CLIP similarity.", "weaknesses": "In image restoration experiments, quantitative evaluation is lacking, as the authors provide only qualitative comparisons with competing methods. Furthermore, the authors do not provide evidence that the reconstruction results produced by their method are consistent with the input measurements. Specifically, it would be helpful to see whether applying the forward measurement operator to the algorithms' output yields results that closely approximate the original measurements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wa4jCkJ26O", "reviewer": "Reviewer_98Bk", "summary": "The paper introduces \"PostEdit,\" a novel inversion- and training-free approach leveraging posterior sampling for efficient zero-shot image editing. The method addresses significant challenges in image editing, including controllability, background preservation, and efficiency. The overall results indicate promising performance, particularly in maintaining high-quality background similarity and efficiency in execution time. Nevertheless, there are a few methodological and experimental aspects that require further clarification and expansion.", "questions": "Please refer to the \"Weaknesses\" section above for my questions and concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method's efficiency in terms of GPU memory and inference time is noteworthy. Achieving high-quality results in approximately 1.5 seconds and with only 18 GB of GPU memory is an impressive step forward for zero-shot image editing.\n2. The authors provide strong mathematical support for their approach, with the incorporation of Langevin dynamics and a posterior measurement term to optimize the estimated image. This theoretically addresses the issue of error accumulation seen in existing approaches.\n3. The results—both qualitative and quantitative—demonstrate the approach's effectiveness, showing its superiority compared to some existing methods.", "weaknesses": "1. Methodological Clarifications:\n\na. Figure 2: Step 3 in Figure 2 is challenging to understand and requires more explanation regarding the optimization process. It would be beneficial to clearly explain how Langevin dynamics and the measurement term are incorporated in this step to improve the optimization.\n\nb. The explanation of Symbols such as $\\mathcal{A}$ are missing: In Equation (16), the symbol $\\mathcal{A}$ is not clearly defined or explained. Adding a clear definition for this symbol would enhance the reader's understanding. Eq.16 needs more explanations for specific symbols such as m, h, and etc.\n\nc. In lines 321-322, the authors mention the mask, however, how to obtain the mask, it seems that the paper without further explanations.\n\n2. Experimental Clarifications:\n\na. Baselines: The experimental comparisons do not include state-of-the-art methods such as SPD Inversion [1], and Guide-and-Rescale [2]. Including these baselines is crucial to substantiate the claims of superiority. \n\nb. Ablation Study: The ablation study lacks quantitative results to further support the analysis. Numerical data on the effectiveness of each component (e.g., posterior sampling vs. inversion process) would add robustness to the conclusions.\n\nc. User Study: Given that image editing quality can be somewhat subjective, a user study would be helpful to establish the practical effectiveness of the proposed method more convincingly.\n\nd. Complex Images: All the reconstruction results shown in the paper involve images with a single object and simple layouts. However, some inversion methods struggle with reconstructing images containing multiple detailed objects. Could you provide reconstruction results for such complex images to strengthen your claims about the method's versatility?\n\n\nReferences for Missing Baselines: \n[1] Li R B, Li R H, Guo S, Zhang L. Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models. European Conference on Computer Vision. 2024.\n\n[2] Titov V, Khalmatova M, Ivanova A, et al. Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing. European Conference on Computer Vision. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9O55jYNEJh", "reviewer": "Reviewer_CdSe", "summary": "This paper introduces PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt.", "weaknesses": "1. The reviewer encountered several points in the algorithm that were difficult to understand, raising the possibility of inaccuracies. The reviewer requests that the authors provide clear responses to the following questions: (1) In line 4, does \\(z_0\\) bear no relation to the original image, given that \\(z_N\\) is derived from pure noise? (2) Can lines 6 to 9 be interpreted as a re-generation of \\(z_0\\)? (3) Is the alignment with the original image (i.e., \\(z_{in}\\)) genuinely implemented only in lines 12 to 14? (4) In line 15, the reviewer finds the superscript \\(N\\) on \\(z^{(N)}\\) unusual, as it would conventionally be \\(m\\) in this context. Additionally, the reviewer is puzzled by the disappearance of \\(z_{\\tau_{i-1}}\\) in the next loop iteration. These uncertainties make the methodology difficult to follow, and the reviewer suggests clarifying these points for future readers.\n\n2. Regarding the essence of the proposed method, the reviewer interprets it as generating an image that aligns with the target prompt, followed by posterior probability correction techniques that reintroduce information from the original image via weighted blending, masking, or similar means. The reviewer has experience with studies on posterior probability and would appreciate confirmation.\n\n3. The reviewer finds the reported speed and runtime of the proposed method particularly perplexing. Given the numerous loops in the algorithm, it is unclear how the method achieves a runtime of only 1.2 seconds. As far as the reviewer is aware, most training-free image editing methods, even with fewer layers in the algorithmic structure, typically take tens of seconds. Could the authors clarify where this dramatic time saving is achieved? Beyond seeing a 1.2-second result, the reviewer would like a breakdown of time components comparing this method to other training-free methods.\n\n4. The number of baseline methods in the comparative experiments is insufficient, with few comparisons to state-of-the-art training-based methods. This lack of comparison feels inequitable, as training-based methods have become well-established for image editing, showing robustness and speed (e.g., InstructP2P, SmartEdit, SEED-X-Edit). The reviewer questions what specific contributions are offered by the proposed training-free method.\n\n5. The proposed method involves an overwhelming number of hyperparameters; there appear to be countless hyperparameters even within the algorithm section alone. The reviewer believes that a detailed hyperparameter sensitivity analysis is necessary to ensure that the model's robustness is not significantly affected by the choice of hyperparameters. From the reviewer's experience, many training-free methods require intricate hyperparameter tuning to achieve optimal results. Therefore, the reviewer requests that the authors provide the specific hyperparameters used for each image in the experimental section.\n\n6. The related work section appears to reference too few relevant studies in the image editing field, which is rapidly evolving. The reviewer suggests expanding this section by referencing additional survey papers to strengthen the literature review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)