PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editing

## Abstract

Abstract In the field of image editing, three core challenges persist: controllability, background preservation, and efficiency. Inversion-based methods rely on time-consuming optimization to preserve the features of the initial images, which results in low efficiency due to the requirement for extensive network inference. Conversely, inversion-free methods lack theoretical support for background similarity, as they circumvent the issue of maintaining initial features to achieve efficiency. As a consequence, none of these methods can achieve both high efficiency and background consistency. To tackle the challenges and the aforementioned disadvantages, we introduce PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt. Extensive experimental results indicate that the proposed PostEdit achieves state-of-the-art editing performance while accurately preserving unedited regions. Furthermore, the method is both inversion- and training-free, necessitating approximately 1.5 seconds and 18 GB of GPU memory to generate high-quality results. Code is
available at https://github.com/TFNTF/PostEdit .

## 1 introduction

Large text-to-image diffusion models Saharia et al. (); Pernias et al. (); Podell et al. (); Ramesh et al. () have demonstrated significant capabilities in generating photorealistic images based on given textual prompts, facilitating both the creation and editing of real images. Current research Cao et al. (); Brack et al. (); Ju et al. (); Parmar et al. (); Wu & la Torre (); Xu et al. () highlights three main challenges in image editing: controllability, background preservation, and efficiency. Specifically, the edited parts must align with the target prompt’s concepts, while unedited regions should remain unchanged. Additionally, the editing process must be sufficiently efficient to support interactive tasks. There are two mainstream categories of image editing approaches, namely inversion-based and inversion-free methods, as illustrated in Fig. 1.

Inversion-based approaches Song et al. (); Mokady et al. (); Wu & la Torre (); Huberman-Spiegelglas et al. () progressively add noise to a clean image and then remove the noise conditioned on a given target prompt, utilizing large text-to-image diffusion models (*i.e*. Stable Diffusion Rombach et al. ()), to obtain the edited image. However, directly inverting the diffusion sampling process (*e.g*., DDIM Song et al. ()) for reconstruction introduces bias from the initial image due to errors accumulated by an unconditional score term, as discussed in classifier-free guidance (CFG) Ho & Salimans () and proven in App. A.1. Consequently, the editing quality of inversion-based methods is primarily constrained by the similarity in unedited regions. Several approaches address this issue by optimizing the text embedding Wu et al. (), employing iterative guidance Kim et al. (); Garibi et al. (), or directly modifying attention layers Hertz et al. (); Mokady et al. (); Parmar et al. () to mitigate the bias introduced by the unconditional term. However, the necessity of adding and subsequently removing noise predicted by a network remains unavoidable, thereby significantly constraining their efficiency. Recent methods Starodubcev et al. (); Li & He (); Kim et al. () attempt to enhance the accuracy of the iterative sampling process by training an invertible consistency trajectory, following the distillation process in the consistency models (CM) Song et al. (); Salimans & Ho (); Song & Dhariwal (); Luo et al. (). Although this approach significantly reduces the accumulation errors from the unconditional term, it cannot eliminate them. Moreover, the editing performance is sensitive to the hyperparameters (*i.e*., the fixed boundary timesteps of multi-step consistency models), and the training process generally demands hundreds of GPU hours.

Another category of methods Brooks et al. (); Mou et al. (); Ye et al. (); Guo et al. (); Li et al. (); Wang et al. () is inversion-free, which omits the inversion process and consequently significantly decreases the inference time. Specifically, these methods follow the approach of ControlNet Zhang et al. () to train a network that embeds additional conditional features, such as hand-drawn outlines, depth maps, and canny maps. However, the works based on ControlNet like T2I-Adapter Mou et al. () highly rely on the accuracy of the input guidance structure while most applications relating to ControlNet are customization. Some other works Zhang et al. (), Zhang et al. (), Hui et al. () employ a diffusion model trained on synthetic edited images, producing edited images in a supervised manner. This methodology obviates the need for inversion process during the sampling stage. Moreover, there is a training-free method to satisfy inversion-free requirement Xu et al. (). It adopts specific settings of the DDIM solver to leverage the advantages of CM to ensure the editing quality. Although these recent works can achieve fast sampling and accurate editing, the aforementioned problem remains unsolved since the diffusion sampling process Ho et al. (); Song et al. (; ) is necessary. Therefore, all the inversion-free methods cannot circumvent the accumulation errors caused by the unconditional score term in CFG.

Figure: Figure 1: Comparison of Different Methods. The process of inversion-based method is shown on the top-left part. It progressively add noise to a clean image by applying the noise estimated by the network and then obtains the edited image by a denoise process. There is an optimization process for maintaining the background similarity. The too-right part is the methods with an inversion-free process, which adds noise randomly and operates training process on network to get satisfied output. Our method is designed to become inversion-free and training-free as shown in the bottom.
Refer to caption: x1.png

In this work, we present an inversion- and training-free method called PostEdit to optimize the accumulated errors of the unconditional term in CFG based on the theory of posterior sampling Kawar et al. (),Chung et al. (), Zhang et al. (), Lugmayr et al. (), Zhu et al. (), Song et al. ().
To reconstruct and edit an image $\boldsymbol{x}_{0}$, we make use of a measurement term $\boldsymbol{y}$ which contains the features of the initial image, and supervise the editing process by the posterior log-likelihood density $\nabla_{\boldsymbol{x}_{t}}\log p(\boldsymbol{x}_{t}|\boldsymbol{y})$.
With this term, we can estimate the target image through progressively sampling from the posterior $p(\boldsymbol{x}_{t}|\boldsymbol{y})$ referring to the Bayes rule.
The above process is reasonable since the inverse problems of probabilistic generative models are ubiquitous in generating tasks, which are trained to learn scores to match gradients of noised data distribution (log density), and this process is also called score matching Song & Ermon (), Song & Ermon (), Karras et al. () and Karras et al. (). $\boldsymbol{y}$ is defined according to the following inverse problem

$$ $\boldsymbol{y}=\mathcal{A}(\boldsymbol{x}_{0})+\boldsymbol{n},$ (1) $$

where $\mathcal{A}$ is a forward measurement operator that can be linear or nonlinear and $\boldsymbol{n}$ is an independent noise. Hence, the posterior sampling strategy can be regarded as a diffusion solver and it can edit images while maintaining the regions that are required to remain unchanged with the measurement $\boldsymbol{y}$. Also, instead of time-consuming training or optimization, our framework adopts an optimization process without requirements for across the network many times for inference, which can be lightweight taking about $1.5$ seconds to operate and around 18 GB of GPU memory. Our contributions and key takeaways are shown as follows:

- •
To the best of our knowledge, we are the first to propose a framework that extends the theory of posterior sampling to text-guided image editing task.
- •
We theoretically address the error accumulation problem by introducing posterior sampling, and designing an inversion-free and training-free strategy to preserve initial features. Furthermore, we replace the step-wise sampling process with a highly efficient optimization procedure, thereby significantly accelerating the overall sampling process.
- •
PostEdit ranks among the fastest zero-shot image editing methods, achieving execution times of less than 2 seconds. Additionally, the state-of-the-art CLIP similarity scores on the PIE benchmark attest to the high editing quality of our method.

## 2 Preliminaries

### 2.1 Score-Based Diffusion Models

A continuous diffusion process as shown in Song et al. () is adopted in this paper to sample the estimated initial image $\boldsymbol{\hat{x}}_{0}$. Specifically, the forward diffusion process can be modeled as the solution to an Itô SDE:

$$ $d\boldsymbol{x}=\boldsymbol{f}_{t}(\boldsymbol{x})dt+g_{t}d\boldsymbol{w},$ (2) $$

where $\boldsymbol{f}$ is defined as the drift function and $g$ denotes the coefficient of noise term. Furthermore, the corresponding reverse form of Eq. 2 can be written as

$$ $d\boldsymbol{x}=\left[\boldsymbol{f}_{t}(\boldsymbol{x})-g_{t}^{2}\nabla_{ \boldsymbol{x}}\log p_{t}(\boldsymbol{x})\right]dt+g_{t}d\boldsymbol{\bar{w}},$ (3) $$

where $\boldsymbol{\bar{w}}$ represents the standard Brownian motion. As shown in Song et al. (), there exists a corresponding deterministic process whose trajectories share the same marginal probability densities as the SDE according to Eq. 2. This deterministic process satisfies an ODE

$$ $d\boldsymbol{x}=\left(\boldsymbol{f}_{t}(\boldsymbol{x})-\frac{1}{2}g_{t}^{2} \nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x})\right)dt.$ (4) $$

The ODE is able to compute the exact likelihood of any input data by leveraging the connection to neural ODEs Chen et al. (). In order to approximate the log density of noised data distribution $\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x})$ for each sampling step, a network Song et al. () $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_{t},t)$ is trained to learn the corresponding log density

$$ $\displaystyle\mathbb{E}_{\boldsymbol{x}_{0},\boldsymbol{x}_{t}\sim p( \boldsymbol{x}_{t}|\boldsymbol{x}_{0})}\left[\left\|\boldsymbol{s}_{ \boldsymbol{\theta}}(\boldsymbol{x}_{t},t)-\nabla_{\boldsymbol{x}_{t}}\log p( \boldsymbol{x}_{t}|\boldsymbol{x}_{0})\right\|^{2}\right].$ (5) $$

### 2.2 DDIM Solver And Consistency Models

The DDIM solver is widely applied in training large text-to-image diffusion models. The iterative scheme for sampling the previous step is defined as follows

$$ $\boldsymbol{x}_{t-1}=\sqrt{\alpha_{t-1}}\left(\frac{\boldsymbol{x}_{t}-\sqrt{1 -\alpha_{t}}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_{t},t)} {\sqrt{\alpha_{t}}}\right)+\sqrt{1-\alpha_{t-1}}\boldsymbol{\epsilon}_{ \boldsymbol{\theta}}(\boldsymbol{x}_{t},t),$ (6) $$

where $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_{t},t)$ is the predicted noise from the network. According to Eq. 6, the sampling process can be regarded as first estimating a clean image $\boldsymbol{x}_{0}$, and then using the forward process of the diffusion models with noise predicted by the network to the previous step $\boldsymbol{x}_{t-1}$. Therefore, the predicted original sample $\boldsymbol{\hat{x}}_{0}$ is defined as

$$ $\boldsymbol{\hat{x}}_{0}=\frac{\boldsymbol{x}_{t}-\sqrt{1-\alpha_{t}} \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_{t},t)}{\sqrt{ \alpha_{t}}}$ (7) $$

Latent consistency models Luo et al. () apply the DDIM solver Song et al. () to predict $\boldsymbol{\hat{x}}_{0}$ and use the self-consistency of an ODE trajectory Song et al. () to distill steps. Then the $\boldsymbol{x}_{0}$ is calculated by the function $\boldsymbol{f}_{\boldsymbol{\theta}}(\boldsymbol{z},\boldsymbol{c},t)$ through large timestep, where $\boldsymbol{f}$ is defined in Eq. 7

$$ $\boldsymbol{f}_{\boldsymbol{\theta}}(\boldsymbol{z},\boldsymbol{c},t)=c_{\text {skip }}(t)\boldsymbol{z}+c_{\text{out }}(t)\left(\frac{\boldsymbol{z}-\sigma_ {t}\epsilon_{\boldsymbol{\theta}}(\boldsymbol{z},\boldsymbol{c},t)}{\alpha_{t} }\right),$ (8) $$

$\boldsymbol{z}$ is denoted as $\boldsymbol{x}$ encoded in the latent space. The loss function of self-consistency is defined as

$$ $\mathcal{L}_{\mathcal{CD}}\left(\boldsymbol{\theta},\boldsymbol{\theta}^{-}; \Psi\right)=\mathbb{E}_{\boldsymbol{z},\boldsymbol{c},n}\left[d\left( \boldsymbol{f}_{\boldsymbol{\theta}}\left(\boldsymbol{z}_{t_{n+1}},\boldsymbol {c},t_{n+1}\right),\boldsymbol{f}_{\boldsymbol{\theta}^{-}}\left(\hat{ \boldsymbol{z}}_{t_{n}}^{\Psi},\boldsymbol{c},t_{n}\right)\right)\right],$ (9) $$

where $\hat{\boldsymbol{z}}_{t_{n}}^{\Psi}$ is an estimation of the evolution of the $\boldsymbol{z}_{t_{n}}$ from $t_{n+1}$ using ODE solver $\Psi$.

### 2.3 Posterior Sampling in Diffusion Models

After obtaining $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_{t},t)$, we can infer an unknown $\boldsymbol{x}\in\mathbb{R}^{d}$ through the degraded measurement $\boldsymbol{y}\in\mathbb{R}^{n}$. Specifically, in the forward process, it is well-posed since the mapping $\boldsymbol{x}\to$ $\boldsymbol{y}:\mathbb{R}^{d}\to\mathbb{R}^{n}$ is many-to-one, while it is ill-posed for the reverse process since it is one-to-many when sampling the posterior $p(\boldsymbol{x}_{0}|\boldsymbol{y})$, where it can not be formulated as a functional relationship. To deal with this problem, the Bayes rule is applied to the log density terms and we can derive that

$$ $\nabla_{\boldsymbol{x}_{t}}\log p(\boldsymbol{x}_{t}|\boldsymbol{y})=\nabla_{ \boldsymbol{x}_{t}}\log p(\boldsymbol{x}_{t})+\nabla_{\boldsymbol{x}_{t}}\log p (\boldsymbol{y}|\boldsymbol{x}_{t}),$ (10) $$

where the first term in the right side hand of the equation is the pre-trained diffusion model and the second one is intractable. The measurement $\boldsymbol{y}$ can be regarded as a vital term that contains the information of the prior $p(\boldsymbol{x})$, which supervises the generation process towards the input images. In order to work out the explicit expression of the second term, existing method DPS Chung et al. () presents the following approximation

$$ $\displaystyle p(\boldsymbol{y}|\boldsymbol{x}_{t})$ $\displaystyle=\mathbb{E}_{\boldsymbol{x}_{0}\sim p(\boldsymbol{x}_{0}| \boldsymbol{x}_{t})}[p(\boldsymbol{y}|\boldsymbol{x}_{0})]\approx p( \boldsymbol{y}|\hat{\boldsymbol{x}}_{0})$ (11) $\displaystyle=\mathbb{E}_{\boldsymbol{x}_{0}\sim p(\boldsymbol{x}_{0}| \boldsymbol{x}_{t})}[\boldsymbol{x}_{0}],$ $$

where the Bayes optimal posterior $\hat{\boldsymbol{x}}_{0}$ can be obtained from a given pre-trained diffusion models or Tweedie’s approach to iterative descent gradient for the case of VP-SDE or DDPM sampling. Hence, each step can be written as $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_{t},\boldsymbol{y})$ according to Eq. 10.

When the transition kernel is defined, since the solvers utilize the unconditional scores to estimate $\hat{\boldsymbol{x}}_{0}$, the measurement term is then introduced through a gradient descent way to optimize $\boldsymbol{x}$

$$ $\boldsymbol{x}_{t-1}=f\left(\boldsymbol{x}_{t},\boldsymbol{\hat{x}}_{0}, \boldsymbol{\epsilon}\right)+\eta\nabla_{\boldsymbol{x}_{t}}\left\|y-\mathcal{ A}(\boldsymbol{\hat{x}}_{0})\right\|^{2}_{2},\ \ \boldsymbol{\epsilon}\sim \mathcal{N}\left(0,\boldsymbol{I}\right),$ (12) $$

where the function $f$ is defined as the approximation of the unconditional counterpart of $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_{t},\boldsymbol{y})$ and $\eta$ denotes the learning rate.

## 3 Method

We propose a new sampling process equipped with a designed optimization process to improve the quality of image reconstruction and editing, and the adopted SDE/ODE solver is based on DDIM shown in Eq. 6. Denote $\boldsymbol{z}\sim\mathcal{E}(\boldsymbol{x}_{0})$, $\boldsymbol{z}\in\mathbb{R}^{p}$ where $\mathcal{E}$ is an encoder and $\boldsymbol{x}_{0}$ is an initial image. Our method operates in latent space and leverages the theory of the posterior sampling to correct the bias from the initial features and introduce the target concepts. The core insight is using the measurements $\boldsymbol{y}$, estimated $\boldsymbol{z}_{0}$ and Langevin dynamics as the optimization terms to correct the errors of the sampling process. The importance of reconstruction and the algorithm are introduced specifically in (Sec. 3.1). The implementation details of the editing process are illustrated in detail (Sec. 3.2). Our method takes around 1.5 seconds and 18 GB memory costs on a single NVIDIA A100 GPU.

### 3.1 Posterior Sampling for Image Reconstruction

The quality of reconstruction is a crucial indicator for evaluating the editing capabilities of a method. To preserve the features of the background (areas unaffected by the target prompt), Mokady et al. () introduces a technique for fine-tuning the text embedding to mitigate errors caused by the null text term, as demonstrated in App. A.1. However, this approach is time-consuming, and there is a pressing need to enhance editing performance. To address this challenge, we propose a method that enables a fast and accurate reconstruction and editing process.

Specifically, there are four steps in our method: (1) We add noise to $\boldsymbol{z}_{0}$ by DDPM noise schedule till $\boldsymbol{z}_{T}\sim\mathcal{N}(0,\boldsymbol{I})$. Compared to the DDIM inversion process applied in Mokady et al. (), the added noise is sampled from $\mathcal{N}(0,\boldsymbol{I})$ so that it is not required to inference with networks many times. Consequently, this process is inversion-free and significantly decreases the time costs. (2) Existing SDE/ODE solvers, such as DDIM and LCM, are applied to obtain an estimated $\boldsymbol{\hat{z}}_{0}$. (3) In order to make $\boldsymbol{\hat{z}}_{0}$ become more consistent with the features of background and target prompt of the original image, it is optimized by two $\mathcal{L}_{2}$ norm terms relating to the defined measurement $\boldsymbol{y}$ and $\boldsymbol{\hat{z}}_{0}$ respectively. Further, the Langevin dynamics is applied to circumvent solutions of local optimal. (4) Finally, by progressively operating the above process to update the mean and variance of the Gaussian distribution in step (1) according to a time schedule, we can obtain the $\boldsymbol{z}_{0}^{*}$ with consistent initial features and accurate target characteristics respectively when $T$ is converged to 0 0. The presented algorithm corresponding to the above process is shown in detail in Fig. 2 for image reconstruction or editing and Alg. A.2 for image reconstruction.

Figure: Figure 2: Method Overview. The latent representation of initial image $\boldsymbol{x}_{0}$ is $\boldsymbol{z}_{0}$. It is adding noise randomly to $\boldsymbol{z}_{T}$ and then $\boldsymbol{\hat{z}}_{0}$ is estimated from $\boldsymbol{z}_{T}$ through diffusion ODE solvers. After that, there are two optimization terms relating to $\boldsymbol{\hat{z}}_{0}$, the given measurement $\boldsymbol{y}$ and a random noise term $\boldsymbol{\epsilon}$, which is applied to optimize calculated $\boldsymbol{\hat{z}}_{0}$ while avoids solutions falling in local optimality. Then the optimized $\boldsymbol{\hat{z}}_{0}$ is adding noise to timestep $T-1$ according to the noise scheduler. This process operates recursively and finished till $\boldsymbol{\hat{z}}_{T}$ is converged to $\boldsymbol{z}_{0}$, where $z_{0}^{*}$ is the finally optimized output.
Refer to caption: x2.png

Our method requires a large text-to-image diffusion model as input and we select Stable Diffusion (SD) Rombach et al. (). Given that SD is trained on the dataset containing billions of images, the generated result has strong randomness relating to the same prompt. Moreover, reconstructing an image with a given text prompt starting from $\mathcal{N}\left(0,\boldsymbol{I}\right)$ usually gets poor results for SD due to the bias caused by the unconditional term in CFG. Therefore, if we directly apply posterior sampling strategy shown in Eq. 12 to acquire $\boldsymbol{z}_{0}$ from $\boldsymbol{z}_{T}\sim\mathcal{N}(0,\boldsymbol{I})$ by leveraging SD to inference noise, $\boldsymbol{\hat{z}}_{0}$ differs greatly from the ground truth $\boldsymbol{z}_{0}$. Conversely, this sampling process has good reconstruction performance when it leverages the diffusion model trained on small datasets, for example, FFHQ Karras et al. () and ImageNet Deng et al. () as shown in Chung et al. (); Zhang et al. (). We experimentally discover that the gap between these two kinds of models is the inconsistent layouts of each estimated image $\boldsymbol{\hat{z}}_{0}$, while the features of the target prompts are successfully introduced into the generated $\boldsymbol{\hat{z}}_{0}$. Specifically, The layouts of $\boldsymbol{\hat{z}}_{0}$ generated by the scores inferred by the networks trained on FFHQ and SD for intermediate timesteps are shown in App. A.5. Therefore, due to the editing and reconstruction trade-off issue, it is much more challenging for high-quality image editing and reconstruction by leveraging large text-to-image models.

To address the editing and reconstruction trade-off issue, we present a weighted process that introduces the features of initial data into the estimated $\boldsymbol{\hat{z}}_{0}$ as shown in the following proposition.

###### Proposition 1 .

The weighted relationship between the estimated $\boldsymbol{\hat{z}}_{0}$ and the initial image $\boldsymbol{z}_{in}$ to correct evaluated $\boldsymbol{z}_{0}$ is defined as $\left(0\leq w\leq 0.1\right)$

$$ $\boldsymbol{z}_{0}^{w}=(1-w)\cdot\boldsymbol{\hat{z}}_{0}+w\cdot\boldsymbol{z} _{in},$ (13) $$

where $w$ is a constant to govern the intensity of the injected features.

Since the accuracy of different solvers are different, the difficulty of image reconstruction is also different. Experiments shown in Sec. 4.4 support that Proposition 1 is important for our method shown in Fig. 2 to be adapted to the widely used DDIM, DDPM, and CM solvers.

###### Remark 1 .

Eq. 13 is reasonable since $(1-w)$ and $w\cdot\boldsymbol{z}_{in}$ are regarded as constant. Hence, this process does not essentially influence the sampling process of distribution $\boldsymbol{z}_{t-1}\sim\mathbb{E}\left(\mathcal{N}\left(\boldsymbol{z}_{0}^{w
},\sigma_{t-1}^{2}\boldsymbol{I}\right)\right)$, which is shown specifically in the Proposition 2.

In order to adapt Eq. 10 to the DDIM solver shown in Eq. 6 adopted by SD, we can derive it as

$$ $\nabla_{\boldsymbol{z}_{0}}\log p(\boldsymbol{z}_{0}|\boldsymbol{z}_{t}, \boldsymbol{y})=\nabla_{\boldsymbol{z}_{0}}\log p(\boldsymbol{z}_{0}| \boldsymbol{z}_{t})+\nabla_{\boldsymbol{z}_{0}}\log p(\boldsymbol{y}| \boldsymbol{z}_{0},\boldsymbol{z}_{t}),$ (14) $$

to calculate the scores towards to $\boldsymbol{z}_{0}$ straightly inspired by Chung et al. () and Zhang et al. (). The measurement settings for image reconstruction are listed in the App. A.4

###### Proposition 2 .

Suppose $\boldsymbol{z}_{t}$ is sampled from time marginal distribution of $p\left(\boldsymbol{z}_{t}|\boldsymbol{y}\right)$, then

$$ $\boldsymbol{z}_{t-1}\sim\mathbb{E}_{\boldsymbol{z}_{0}^{w}}{\mathcal{N}}\left( \boldsymbol{z}_{0}^{w},\sigma_{t-1}^{2}\boldsymbol{I}\right)$ (15) $$

satisfies the time marginal distribution conditioned on $p\left(\boldsymbol{z}_{t-1}|\boldsymbol{y}\right)$, where $\boldsymbol{z}_{0}^{w}$ is obtained from Eq. 13. (Proof is shown in Appendix. A.3)

Propsition 2 ensures that $\boldsymbol{z}_{t-1}$ sampled from the Gaussian distribution with mean of $\boldsymbol{z}_{0}^{w}$ and variance of $\sigma_{t-1}^{2}$ respectively still satisfies the constraint of the posterior sampling Eq. 10. Therefore, we can present the following scheme to optimize the estimated $\boldsymbol{\hat{z}}_{0}$ and run Langevin dynamics Welling & Teh ()

$$ $\boldsymbol{z}_{0}^{(k+1)}=(1-w)\cdot\boldsymbol{z}_{0}^{(k)}+w\cdot \boldsymbol{z}_{in}-h\cdot\left(\frac{\|z_{0}^{(k)}-z_{0}\|^{2}}{2\sigma_{t}^{ 2}}+\frac{\|\mathcal{A}\left(\boldsymbol{z}_{0}^{(k)}\right)-\boldsymbol{y}\|^ {2}}{2m^{2}}\right)+\sqrt{2h}\epsilon,\ \ \epsilon\sim\mathcal{N}(\boldsymbol{ 0},\mathcal{I}).$ (16) $$

Experimental results in Fig 10 in Appendix. A.7 reflect that we can add noise randomly or completely sample from $\mathcal{N}\left(0,\boldsymbol{I}\right)$ to obtain a high quality reconstruction outcomes without any tuning process by our method. Eq. 16 is reasonable since the two terms that multiplied by the step size $h$
have the same descent direction towards to the ground truth $\boldsymbol{z}_{0}$. Additionally, the Langevin dynamics is applied to search for solutions with global optimal. Since the step (1) shown in Fig. 2 is different from the process of transitioning from the initial image to noise , which involves adding noise $\boldsymbol{\epsilon}_{\theta}\left(\boldsymbol{z}_{t},t,c_{ini}\right)$ inferred by the network at each step Mokady et al. () (where $c_{ini}$ represents the prompt describing the content of the initial image). PostEdit is much more efficient as mentioned before. Considering the fact that there is no information relating to the initial image added to the noised distribution, the term containing the measurement $\boldsymbol{y}$ defined in Eq. 16 is introduced to correct the errors from the initial features caused by the unconditional term shown in CFG, ensuring background similarity.

The detailed process is outlined in App. A.2. As mentioned earlier, Proposition 1 is introduced due to the flexible layouts of the images generated by stable diffusion, which are not essential for the editing task. Additionally, the implementation details are provided in Sec. 3.2, following the introduction of more parameters.

###### Proposition 1 .

###### Remark 1 .

###### Proposition 2 .

### 3.2 Posterior Sampling for Image Editing

In this section, we provide a detailed illustration of the sampling process for high-quality image editing using the DDIM solver Luo et al. (), as shown in Eq. 6. Unlike the ODE solver and measurement $\boldsymbol{y}$ used in the image reconstruction task described in Sec. 3.1, image editing requires a masked measurement derived from $\boldsymbol{z}_{0}$ and an ODE solver with higher accuracy to estimate $\boldsymbol{\hat{z}}_{0}$. The measurement $\boldsymbol{y}$ for image editing is defined as

$$ $\boldsymbol{y}\sim\mathcal{N}\left(\boldsymbol{P}\boldsymbol{z},\sigma^{2} \boldsymbol{I}\right),$ (17) $$

where $\boldsymbol{P}\in\{0,1\}^{n\times p}$ represents a masking matrix composed of elementary unit vectors. This measurement setup not only serves as a specialized configuration for image editing but also demonstrates its capacity to deliver high-quality image reconstruction results, even when $\boldsymbol{z}_{0}$ is masked. The settings in Eq.17 have been validated to yield high-quality reconstruction results, as shown in Sec.4.4, providing strong evidence of the method’s ability to preserve the features of the initial image.

Furthermore, to minimize the number of sampling steps, improving the accuracy of the estimated $\boldsymbol{\hat{z}}_{0}$ is crucial. We employ the LCM solver Luo et al. () to predict $\boldsymbol{\hat{z}}_{0}$ from the noised input image $\boldsymbol{z}_{N}$ less than 4 steps, where the LCM is distilled from the models based on the DDIM solver. Experimental results show that the superior denoising capabilities of the LCM solver significantly accelerate the convergence rate and produce more accurate $\boldsymbol{\hat{z}}_{0}$ estimates that align with the target prompt.
The measurement characteristics $\boldsymbol{y}$, as defined in Eq.17, involve randomly masking each element of $\boldsymbol{z}_{0}$ with a probability $p$. Since one of the optimization terms focuses on only a small portion of the initial image, both terms in Eq.16 guide the gradient descent in the same direction. As the sampling process progresses, the edited $\boldsymbol{x}_{tgt}$ gradually inherits features from both $\boldsymbol{x}_{0}$ and the target prompt by selectively replacing the necessary attributes.
The experimental results of different settings for the optimization defined in Eq.16 are presented in Sec.4.4. The rest of the process mirrors the reconstruction phase, allowing us to progressively achieve the edited $\boldsymbol{x}_{0}$. In summary, the algorithm’s procedure is detailed in Alg. 1, with implementation specifics provided in App.A.4.

Figure: Algorithm 1 Posterior Sampling for Image Editing

## 4 Experiments

### 4.1 Experiment Setup

To ensure a fair comparison, all experiments were conducted on the PIE-Bench dataset Ju et al. () using a single A100 GPU to evaluate both image quality and inference efficiency. The PIE-Bench dataset comprises 700 images with 10 types of editing, where each image is paired with a source prompt and a target prompt. In our experiments, the resolution of all test images was set to $512\times 512$. For the reconstruction experiments, we set the initial and target prompts to be identical across all test runs. Additional settings are detailed in App. A.4.

### 4.2 Quantitative Comparison

To evaluate the performance of our proposed method, a quantitative comparison is presented in Tab. 1. We compare our method against seven recent image editing approaches: NTIMokady et al. (), NPI Miyake et al. (), PnP Tumanyan et al. (), DI Ju et al. (), iCD Starodubcev et al. (), DDCM Xu et al. (), and TurboEdit Deutch et al. (). The comparison is based on three key metrics: background consistency, CLIP Radford et al. () similarity, and efficiency, which are used as comparative benchmarks in our evaluation.

**Table 1: Quantitative Comparisons of Editing. Results are evaluated on eight different methods, covering the evaluation of background similarity and editing performance. $\boldsymbol{*}$ means this model benefits from the inference superiority of the SDXL-Turbo model and the speed of our method can be further improved since we adopt LCM-SD1.5 as the baseline model.**
| Method | Background Preservation | CLIP Similarity | Efficiency |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | LPIPS${}_{\times 10^{2}}$ | MSE${}_{\times 10^{3}}$ | SSIM${}_{\times 10^{2}}$ | Whole | Edited | Time |  |
| $\uparrow$ | $\downarrow$ | $\downarrow$ | $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ |  |
| NTI | 27.50 | 5.67 | 3.40 | 85.03 | 25.08 | 21.36 | $\sim$ 120s |
| NPI | 25.81 | 7.48 | 4.34 | 83.44 | 25.52 | 22.24 | $\sim$ 15s |
| PnP | 22.31 | 11.29 | 8.31 | 79.61 | 25.92 | 22.65 | $\sim$ 240s |
| DI | 27.28 | 5.38 | 3.25 | 85.34 | 25.71 | 22.17 | $\sim$ 60s |
| iCD | 22.80 | 10.30 | 7.96 | 79.44 | 25.61 | 22.33 | $\sim$ 1.8s |
| DDCM | 28.08 | 5.61 | 7.06 | 85.26 | 26.07 | 22.09 | $\sim$ 2s |
| TurboEdit | 22.44 | 10.36 | 9.51 | 80.15 | 26.29 | 23.05 | $\sim$ 1.2s^∗ |
| Ours | 27.04 | 6.38 | 3.24 | 82.20 | 26.76 | 24.14 | $\sim$ 1.5s |

The experimental results reflect that PostEdit achieves SOTA performance on editing, which are the “Whole” and “Edited” indicators of the CLIP similarity and the results are significantly better than others. For efficiency, our model is highly efficient with a runtime less than 2 seconds. It is worth noting that our runtime is slightly higher than TurboEdit Deutch et al. (), which is mainly due to different baselines. Specifically, TurboEdit employs SDXL-Turbo Sauer et al. () while our framework is based on LCM-SD1.5 Luo et al. (). As shown in App. A.6, SDXL-Turbo Sauer et al. () is almost $2.5$ times faster than LCM-SD1.5 Luo et al. (). We believe the efficiency of our framework can be further improved if we adopt a more efficient baseline like SDXL-Turbo. For the aspect of background preservation, our method possesses the best result on MSE. In the following part, more qualitative results can further demonstrate the superiority of our framework in background preservation.

In conclusion, our method demonstrates superior editing capability and efficiency compared to the selected recent approaches, while also effectively preserving background similarity.

### 4.3 Qualitative Comparison of Reconstruction and Editing

To demonstrate the superiority of PostEdit in image reconstruction, we present relevant results in Fig. 3. The experiments indicate that our method exhibits the most stable reconstruction capacity and high quality generation ability compared to NTI Mokady et al. (), NPI Miyake et al. () and icd Starodubcev et al. (), where there is at least one failure case produced by their method. Specifically, the inversion-free method DDCM Xu et al. () fails to faithfully reconstruct the input images, supporting our claim made in Sec. 1. While other methods yield better results in the given cases, they require significantly longer processing times to achieve competitive outcomes. Therefore, our approach offers the best overall performance when considering inference efficiency, stability in generation, and image quality. More reconstruction results are shown in Fig. 10.

To intuitively compare the editing capability, we present a qualitative comparison of eight methods. As illustrated in Fig. 4, PostEdit most effectively highlights the features present in the target prompt, aligning with the quantitative results shown in Tab. 1. Furthermore, the visualized experiments indicate that our method successfully preserves the initial features. Additional comparison results can be found in App. A.7.

Figure: Figure 3: Qualitative Comparison of Reconstruction. It takes 1.5 seconds for our method to reconstruct the input image, and the time is 1.8s, 2s, 15s, and 120s for iCD, DDCM, NPI, and NTI, respectively. Our framework can faithfully reconstruct the foreground object and the background.
Refer to caption: x3.png

Figure: Figure 4: Qualitative Comparison of Editing. Our method performs better than the others in aligning with target prompts while maintaining the background similarity.
Refer to caption: x4.png

### 4.4 Ablation Study

In this section, we conduct various ablation studies and present the results to demonstrate the effectiveness of our framework. (a) We remove the optimization component shown in Eq. 16 and directly apply the adopted SDE/ODE solver to estimate $\boldsymbol{x}_{0}$. The experimental results indicate that the edited images lack background preservation. For instance, in the slanted bicycle example shown in the first row of Fig.5, the staircase on the left side of the original image is transformed into a car in the edited image. (b) We modify the masked probability of our measurement $\boldsymbol{y}$. Notably, there is no discernible difference between the edited images and the input images. (c) We investigate the influence of Proposition 1 on the experimental outcomes, which highlights the effectiveness of the parameter $w$ concerning background similarities.

Figure: Figure 5: Ablation Study Relating to Our Settings. We show the results without the optimization process shown in Eq. 16, the measurement $\boldsymbol{y}$ defined as a mask matrix and $\boldsymbol{z}_{in}$ shown in Proposition 1.
Refer to caption: x5.png

## 5 Conclusion And Limitation

In this work, we address the errors caused by the unconditional term in Classifier-Free Guidance (CFG) by introducing the theory of posterior sampling to enhance reconstruction quality for image editing. To the best of our knowledge, our work is the first to integrate the theory of posterior sampling into the image editing task based on large text-to-image diffusion models. By minimizing the need for repeated network inference, our method demonstrates fast and accurate performance while effectively preserving background similarity, as evidenced by the results. Ultimately, our approach tackles three key challenges associated with image editing and showcases state-of-the-art (SOTA) performance in terms of editing capabilities and inference speed.

Limitation: Since this method is conditioned on a given text prompt, it can be challenging to represent specific scenes. For example, describing ”a man raising his hand” is considerably more difficult compared to the input format used in ControlNet-related works. Additionally, the quality and speed of generation are contingent upon the baseline models. We are actively considering extending our method to incorporate SDXL-turbo or other state-of-the-art models in the future.

## Appendix A Appendix

### A.1 Classifier free diffusion guidance

According to CFG Ho & Salimans (), the generation process is governed by the conditional score, which can be derived as follows

$$ $\displaystyle\nabla_{\boldsymbol{x}_{t}}\log p(\boldsymbol{x}_{t}\mid \boldsymbol{c})=$ $\displaystyle\nabla_{\boldsymbol{x}_{t}}\log\left(\frac{p\left(\boldsymbol{x}_ {t}\right)p(c\mid\boldsymbol{x}_{t})}{p(c)}\right)$ (18) $\displaystyle=$ $\displaystyle\nabla_{\boldsymbol{x}_{t}}\log p(\boldsymbol{x}_{t})+p(c\mid \boldsymbol{x}_{t})$ $\displaystyle-\nabla_{\boldsymbol{x}_{t}}\log p(c)$ $\displaystyle=$ $\displaystyle\nabla_{\boldsymbol{x}_{t}}\log p(\boldsymbol{x}_{t})+\nabla_{ \boldsymbol{x}_{t}}\log p(c\mid\boldsymbol{x}_{t}).$ $$

And then the term $\nabla_{\boldsymbol{x}_{t}}\log p(c\mid\boldsymbol{x}_{t})$ can be derived as

$$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log p\left(c\mid\mathbf{x}_{t}\right)$ $\displaystyle=\nabla_{\mathbf{x}_{t}}\log p\left(\mathbf{x}_{t}\mid c\right)- \nabla_{\mathbf{x}_{t}}\log p\left(\mathbf{x}_{t}\right)$ (19) $\displaystyle=-\frac{1}{\sqrt{1-\bar{\alpha}_{t}}}\left(\boldsymbol{\epsilon}_ {\theta}\left(\mathbf{x}_{t},t,c\right)-\boldsymbol{\epsilon}_{\theta}\left( \mathbf{x}_{t},t\right)\right).$ $$

Substituting the above term into the gradients of classifier guidance, we can obtain

$$ $\displaystyle\overline{\boldsymbol{\epsilon}}_{\theta}\left(\mathbf{x}_{t},t,c\right)$ $\displaystyle=\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,c\right)- \sqrt{1-\bar{\alpha}_{t}}w\nabla_{\mathbf{x}_{t}}\log p\left(c\mid\mathbf{x}_{ t}\right)$ (20) $\displaystyle=\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,c\right)+w \left(\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,c\right)- \boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t\right)\right)$ $\displaystyle=(w+1)\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,c \right)-w\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t\right).$ $$

Clearly, there is an unconditional term (also known as the null-text term) that directly contributes to the bias in the estimation of $\boldsymbol{x}_{0}$ when the DDIM inversion process is applied under Classifier-Free Guidance (CFG) conditions. To mitigate this influence, a tuning process is typically required to optimize the null-text term, ensuring high-quality reconstruction. Furthermore, to achieve a better alignment between the generated image and the text prompt, as well as to enhance image quality, it is often necessary to utilize a larger value of $w$. However, this can exacerbate cumulative errors, leading to significant deviations in the acquired latent representation.

### A.2 Algorithm for Image Reconstruction

The overall process of image reconstruction by applying posterior sampling is shown specifically in Alg. 2.

Figure: Algorithm 2 : Posterior Sampling for Image Reconstruction

The main difference between the reconstruction and editing tasks is the prompt applied to estimate $\boldsymbol{\hat{z}}_{0}$, where it is the initial and target prompts remain the same with each other for image reconstruction.

### A.3 Proof of Proposition 2

According to Eq. 14, the distribution of $\boldsymbol{z}_{t-1}$ depends on $\boldsymbol{z}_{t}$ and $\boldsymbol{z}_{0}$. The marginal distribution relating to timestep $t-1$ can be rewritten by
Proof. We first factorize the measurement conditioned time-marginal $p\left(\mathbf{z}_{t_{2}}\mid\mathbf{y}\right)$ by

$$ $\displaystyle p\left(\mathbf{z}_{t-1}\mid\mathbf{y},c\right)$ $\displaystyle=\iint p\left(\mathbf{z}_{t-1},\mathbf{z}_{0}^{w},\mathbf{z}_{t} \mid\mathbf{y}\right)\mathrm{d}\mathbf{z}_{0}^{w}\mathrm{~{}d}\mathbf{z}_{t}$ (21) $\displaystyle=\iint p\left(\mathbf{z}_{t}\mid\mathbf{y},c\right)p\left(\mathbf {z}_{0}^{w}\mid\mathbf{z}_{t},\mathbf{y},c\right)p\left(\mathbf{z}_{t-1}\mid \mathbf{z}_{0}^{w},\mathbf{z}_{t},y,c\right)\mathrm{d}\mathbf{z}_{0}^{w} \mathrm{~{}d}\mathbf{z}_{t},$ $$

according to the proposition 1, the above equation can be written as

$$ $\displaystyle p\left(\mathbf{z}_{t-1}\mid\mathbf{y},c\right)$ $\displaystyle=\iint p\left(\mathbf{z}_{t}\mid\mathbf{y},c\right)p\left(\mathbf {z}_{0}^{w}\mid\mathbf{z}_{t},\mathbf{y},c\right)p\left(\mathbf{z}_{t-1}\mid \mathbf{z}_{0}^{w},\mathbf{z}_{t},\mathbf{y},c\right)\mathrm{d}\mathbf{z}_{0}^ {w}\mathrm{~{}d}\mathbf{z}_{t}$ (22) $\displaystyle=\iint p\left(\mathbf{z}_{t}\mid\mathbf{y},c\right)p\left(\left(( 1-w)\cdot\mathbf{z}_{0}+w\cdot\mathbf{z}_{in}\right)\mid\mathbf{z}_{t},\mathbf {y},c\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ p\left(\mathbf{z}_{t-1}\mid\left((1-w) \cdot\mathbf{z}_{0}+w\cdot\mathbf{z}_{in}\right),\mathbf{z}_{t},y,c\right) \mathrm{d}\left((1-w)\cdot\mathbf{z}_{0}+w\cdot\mathbf{z}_{in}\right)\mathrm{~ {}d}\mathbf{z}_{t}$ $\displaystyle\overset{\text{(i)}}{=}\iint p\left(\mathbf{z}_{t}\mid\mathbf{y} \right)\left[\left(1-w\right)\cdot p\left(\mathbf{z}_{0}\mid\mathbf{z}_{t}, \mathbf{y},c\right)+w\cdot p\left(\mathbf{z}_{in}\mid\mathbf{z}_{t},\mathbf{y} \right)\right]$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ p\left(\mathbf{z}_{t-1}\mid(1-w)\cdot \mathbf{z}_{0},\mathbf{z}_{t},y\right)\mathrm{d}\left((1-w)\cdot\mathbf{z}_{0} +w\cdot\mathbf{z}_{in}\right)\mathrm{~{}d}\mathbf{z}_{t}$ $\displaystyle\overset{\text{(i)}}{=}\iint p\left(\mathbf{z}_{t}\mid\mathbf{y} \right)p\left(\left(1-w\right)\cdot\mathbf{z}_{0}\mid\mathbf{z}_{t},\mathbf{y} ,c\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ p\left(\mathbf{z}_{t-1}\mid(1-w)\cdot \mathbf{z}_{0},\mathbf{z}_{t},y\right)\mathrm{d}\left((1-w)\cdot\mathbf{z}_{0} \right)\mathrm{~{}d}\mathbf{z}_{t}$ $\displaystyle\overset{\text{(ii)}}{=}\iint p\left(\mathbf{z}_{t}\mid\mathbf{y} \right)p\left(\mathbf{z}_{0}\mid\mathbf{z}_{t},\mathbf{y},c\right)p\left( \mathbf{z}_{t-1}\mid\mathbf{z}_{0},\mathbf{z}_{t},y\right)\mathrm{d}\mathbf{z} _{0}\mathrm{~{}d}\mathbf{z}_{t}$ $\displaystyle=\iint p\left(\mathbf{z}_{t}\mid\mathbf{y}\right)p\left(\mathbf{z }_{0}\mid\mathbf{z}_{t},\mathbf{y},c\right)p\left(\mathbf{z}_{t-1}\mid\mathbf{ z}_{0}\right)\mathrm{d}\mathbf{z}_{0}\mathrm{~{}d}\mathbf{z}_{t}$ $\displaystyle=\mathbb{E}_{\mathbf{z}_{t}\sim p\left(\mathbf{z}_{t}\mid\mathbf{ y}\right)}\mathbb{E}_{\mathbf{z}_{0}\sim p\left(\mathbf{z}_{0}\mid\mathbf{z}_{ t_{1}},\mathbf{y},c\right)}p\left(\mathbf{z}_{t-1}\mid\mathbf{z}_{0}\right)$ $\displaystyle\overset{\text{(iii)}}{=}\mathbb{E}_{\mathbf{z}_{0}\sim p\left( \mathbf{z}_{0}\mid\mathbf{z}_{t},\mathbf{y},c\right)}\mathcal{N}\left(\mathbf{ z}_{t-1};\mathbf{z}_{0},\sigma_{t-1}^{2}\boldsymbol{I}\right),$ $$

where (i) is dues to independent relationships and (ii) is derived by variable substitution and $c$ is the given target prompt. (iii) is derived directly according to the process defined in Eq. 15, whose independent variant is substituted by $\boldsymbol{z}_{0}$ instead of $\boldsymbol{z}_{0}^{w}$.

### A.4 Implementation Details

GPU. All our experiments are performed on a single NVIDIA A100 GPU with 80GB memory.

Parameters in Alg. 1. $N$ is set to $5$ for schedule $\{\tau_{i}\}_{i=1}^{N}$. For better quality of editing, we sometimes sample $\boldsymbol{z}_{N}$ by

$$ $\boldsymbol{z}_{N}\sim\mathcal{N}\left(\sqrt{\bar{\alpha}_{t}}\boldsymbol{z}_{ 0},\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{I}\right),$ (23) $$

where $t$ is set to $501$ generally according to the DDPM schedule Ho et al. (). The reconstruction process is not restricted to this setting. For Eq. 16, $m$ is set as $0.01$ for both the reconstruction and editing task while $\sigma_{t}$ can be referred to the corresponding timestep of DDPM scheduler Ho et al. (). Generally, we apply Eq. 8 for $1$ step to estimate $\boldsymbol{z}_{0}$, and then according to the following schedule to make $\boldsymbol{z}_{\tau_{i}}$ progressively converge to $\boldsymbol{z}_{0}$.

$$ $\{\tau_{i}\}_{i=1}^{5}=\{501,401,301,201,101,1\}.$ (24) $$

The parameter $w$ is usually set to a minimal value such as $0.05$ for most cases or 0 0 and $0.1$ for easy and hard cases. Additionally, $h$ is always set to 1e-5 for image edting and reconstruction tasks.

Parameters of Consistency models. $k_{skip}$ and $k_{out}$ shown in Eq. 8 are set to 0 and 1 for most cases respectively.

ODE Solvers. We adopt the solver of LCM Luo et al. () distilled from Dreamshaper v7 fine-tune of Stable-Diffusion v1-5 for images editing task. For reconstruction, different solvers, for instance, DDIM Song et al. (), DDPM Ho et al. (), and Song et al. () based on Stable Diffusion Rombach et al. () are able to work out satisfied reconstruction quality.

Oversampling Matrix. We set $\sigma$ shown in Eq. 27 to $0.01$ and use an oversampling factor $k=2$ and $n=8$.

2D Discrete Fourier Transform Matrix. The 2D Fourier transform is defined as

$$ $\displaystyle F\left[u,v\right]=\frac{1}{\sqrt{MN}}\sum_{x=0}^{M-1}\sum_{y=0}^ {N-1}f(x,y)\exp\left[{-j2\pi\left(\frac{xu}{M}+\frac{yv}{N}\right)}\right],$ (25) $\displaystyle u=0,1,...,M-1;\ \ v=0,1,...,N-1,$ $$

where $f(x,y)$ is denoted as a two-dimensional discrete signal with dimension $M\times N$ obtained by sampling at superior intervals in the spatial domain. $x$ and $y$ are discrete real variables and discrete frequency variables, respectively. In this paper, the $\boldsymbol{z}_{0}$ is represented as a 2D matrix and operated according to Eq. 25.

Probability of Masked Features. We use the probability equal to 0.5 for a randomly mask process, which represents whether one of the latent features is masked or not.

Measurement $\boldsymbol{y}$ Used for Image Reconstruction. The measurement $\boldsymbol{y}$ can be chosen from any of the following two, which are defined as linear and nonlinear operations relating to initial image $\boldsymbol{z}_{0}$ in latent space

$$ $\boldsymbol{y}\sim\mathcal{N}\left(\boldsymbol{P}\boldsymbol{z},\sigma^{2} \boldsymbol{I}\right),$ (26) $$

$$ $\boldsymbol{y}\sim\mathcal{N}\left(|\boldsymbol{FP}\boldsymbol{z}_{0}|,\sigma^ {2}\boldsymbol{I}\right),$ (27) $$

where $\boldsymbol{F}$ and $\boldsymbol{P}$ denote the 2D discrete Fourier transform matrix and the oversampling matrix with ratio $k/n$ respectively for Eq. 27.

SD Model. We adopt LCM-SD1.5 for all the experiments Luo et al. ().

FFHQ Model. We adopt the ffhq_10m.pt with a size of 357.1MB as the baseline model for all the experiments relating to the FFHQ dataset.

### A.5 Comparison The Images Layout of Different Datasets

In this section, we present a comparison of the layouts of the estimated $\boldsymbol{x}_{0}$ at different intermediate timesteps, as inferred by the diffusion models trained on the SD and FFHQ datasets respectively.

Figure: Figure 6: Layouts Comparison of Evaluated Output for the same Objects in Different Intermediate Timesteps.
Refer to caption: x6.png

Figure: Figure 7: Layouts of Evaluated Output for Different Objects and Timesteps.
Refer to caption: x7.png

In Fig. 6, we present three independent sets of results for both SD and FFHQ, each containing nine different instances of $\boldsymbol{\hat{z}}_{0}$ selected from outputs of various iterations. The first three rows display the results for SD, while the remaining rows correspond to FFHQ. Each set is tasked with generating the same target image based on the same initial image. From left to right, the level of noise progressively decreases.

Notably, the layouts for SD are more varied, with inconsistencies in the cat’s appearance, its position relative to the mirror, and the mirror’s appearance across the three images. This contrasts sharply with the results from FFHQ, where the layouts consistently feature a centered face surrounded by a stable background.

To verify that this property is consistently observed in results based on the FFHQ model, we present additional examples in Fig. 7. As we move from bottom to top, the noise gradually decreases, while from left to right, there are 10 different examples. Each image represents the estimated $\boldsymbol{z}_{0}$ from different iterations.

### A.6 Comparison Between LCM-SD1.5 And SDXL-Turbo

Fig. 8 illustrates the inference speed of LCM-SD1.5, which is utilized in our method, alongside SDXL-Turbo. The results indicate that TurboEdit Deutch et al. () may not be faster than our method, despite its reliance on the advanced baseline model, SDXL-Turbo. All experiments were conducted on a single NVIDIA A100 GPU with 80GB of memory.

Figure: Figure 8: Comparison of Inference Speed.
Refer to caption: x8.png

### A.7 Additional Results

The additional experimental results are exhibited here.

Figure: Figure 9: Additional Comparison Results.
Refer to caption: x9.png

Figure: Figure 10: Additional reconstruction Results.
Refer to caption: x10.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the Weaknesses part.
soundness: 3
strengths: 1. By extending the theory of posterior sampling to text-guided image editing tasks, the proposed method, PostEdit, eliminates the need for both inversion and training. 

2. PostEdit is one of the fastest zero-shot image editing approaches, achieving execution times of under 2 seconds on an A100 GPU.
summary: This paper tackles the issues caused by the unconditional term in Classifier-Free Guidance (CFG) by integrating the theory of posterior sampling to improve reconstruction quality for image editing. By reducing the reliance on repeated network inference, the proposed method, PostEdit, achieves fast and accurate performance while successfully maintaining background consistency. The results seem promising.
weaknesses: While the paper presents an interesting approach, some details and experimental results are not sufficiently comprehensive. Key aspects of the methodology are not fully elaborated, and additional experiments would be beneficial to further validate the claims. 

1. The subfigure "Our Posterior Sampling Process" in Fig. 1 is difficult to understand. It is unclear what exactly it is meant to represent. Additionally, how does it highlight the advantages of the proposed algorithm?

2. The authors mention that the proposed algorithm runs in less than 2 seconds, but it is not clear how many denoising steps are actually performed. In Algorithm 1, the values of N and n are not specified. These details are crucial because if the number of denoising steps is high, it could naturally lead to slower execution. It would also be beneficial to visualize some intermediate results for better clarity.

3. Regarding the editing process, how is the mask generated and provided? There is a lack of details on this aspect.

4. The current editing experiments only involve replacing a few words in the prompt, which seems quite limited. Does the method support more extensive editing operations? More explanation and additional experiments would be appreciated.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - In equation 16, the gradient symbol is missing before the large brackets. The same issue occurs in line 13 of Algorithm 1 and row 710 in Algorithm 2.

- For a more comprehensive background, it may be beneficial to include a citation to [1], which introduced a method for solving linear image restoration problems in cases with noisy observations.

[1] Kawar, Bahjat, Gregory Vaksman, and Michael Elad. "SNIPS: Solving noisy inverse problems stochastically." Advances in Neural Information Processing Systems 34 (2021): 21757-21769.
soundness: 3
strengths: The proposed algorithm is both inversion-free and training-free, contributing to its fast performance. The manuscript is well-written and clear. In editing experiments, the algorithm outperforms state-of-the-art approaches in CLIP similarity.
summary: This paper introduces a diffusion-based method designed for image reconstruction and editing, which operates within an encoded latent space. The proposed Langevin update rule for the latent vector at time $t+1$ consists of three components: a convex combination of the initial clean latent vector and the latent vector at time $t$, a negative gradient sum that guides the vector toward the initial noisy latent vector, and the measurement, and added Gaussian noise. 

The authors evaluate the proposed method through image reconstruction and editing experiments, providing qualitative and quantitative comparisons with state-of-the-art approaches. In editing experiments, the method outperforms the competitors in CLIP Similarity and achieves competitive results in other metrics while also being one of the fastest methods.
weaknesses: In image restoration experiments, quantitative evaluation is lacking, as the authors provide only qualitative comparisons with competing methods. Furthermore, the authors do not provide evidence that the reconstruction results produced by their method are consistent with the input measurements. Specifically, it would be helpful to see whether applying the forward measurement operator to the algorithms' output yields results that closely approximate the original measurements.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the "Weaknesses" section above for my questions and concerns.
soundness: 3
strengths: 1. The proposed method's efficiency in terms of GPU memory and inference time is noteworthy. Achieving high-quality results in approximately 1.5 seconds and with only 18 GB of GPU memory is an impressive step forward for zero-shot image editing.
2. The authors provide strong mathematical support for their approach, with the incorporation of Langevin dynamics and a posterior measurement term to optimize the estimated image. This theoretically addresses the issue of error accumulation seen in existing approaches.
3. The results—both qualitative and quantitative—demonstrate the approach's effectiveness, showing its superiority compared to some existing methods.
summary: The paper introduces "PostEdit," a novel inversion- and training-free approach leveraging posterior sampling for efficient zero-shot image editing. The method addresses significant challenges in image editing, including controllability, background preservation, and efficiency. The overall results indicate promising performance, particularly in maintaining high-quality background similarity and efficiency in execution time. Nevertheless, there are a few methodological and experimental aspects that require further clarification and expansion.
weaknesses: 1. Methodological Clarifications:

a. Figure 2: Step 3 in Figure 2 is challenging to understand and requires more explanation regarding the optimization process. It would be beneficial to clearly explain how Langevin dynamics and the measurement term are incorporated in this step to improve the optimization.

b. The explanation of Symbols such as $\mathcal{A}$ are missing: In Equation (16), the symbol $\mathcal{A}$ is not clearly defined or explained. Adding a clear definition for this symbol would enhance the reader's understanding. Eq.16 needs more explanations for specific symbols such as m, h, and etc.

c. In lines 321-322, the authors mention the mask, however, how to obtain the mask, it seems that the paper without further explanations.

2. Experimental Clarifications:

a. Baselines: The experimental comparisons do not include state-of-the-art methods such as SPD Inversion [1], and Guide-and-Rescale [2]. Including these baselines is crucial to substantiate the claims of superiority. 

b. Ablation Study: The ablation study lacks quantitative results to further support the analysis. Numerical data on the effectiveness of each component (e.g., posterior sampling vs. inversion process) would add robustness to the conclusions.

c. User Study: Given that image editing quality can be somewhat subjective, a user study would be helpful to establish the practical effectiveness of the proposed method more convincingly.

d. Complex Images: All the reconstruction results shown in the paper involve images with a single object and simple layouts. However, some inversion methods struggle with reconstructing images containing multiple detailed objects. Could you provide reconstruction results for such complex images to strengthen your claims about the method's versatility?


References for Missing Baselines: 
[1] Li R B, Li R H, Guo S, Zhang L. Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models. European Conference on Computer Vision. 2024.

[2] Titov V, Khalmatova M, Ivanova A, et al. Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing. European Conference on Computer Vision. 2024.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: This paper introduces PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt.
summary: This paper introduces PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt.
weaknesses: 1. The reviewer encountered several points in the algorithm that were difficult to understand, raising the possibility of inaccuracies. The reviewer requests that the authors provide clear responses to the following questions: (1) In line 4, does \(z_0\) bear no relation to the original image, given that \(z_N\) is derived from pure noise? (2) Can lines 6 to 9 be interpreted as a re-generation of \(z_0\)? (3) Is the alignment with the original image (i.e., \(z_{in}\)) genuinely implemented only in lines 12 to 14? (4) In line 15, the reviewer finds the superscript \(N\) on \(z^{(N)}\) unusual, as it would conventionally be \(m\) in this context. Additionally, the reviewer is puzzled by the disappearance of \(z_{\tau_{i-1}}\) in the next loop iteration. These uncertainties make the methodology difficult to follow, and the reviewer suggests clarifying these points for future readers.

2. Regarding the essence of the proposed method, the reviewer interprets it as generating an image that aligns with the target prompt, followed by posterior probability correction techniques that reintroduce information from the original image via weighted blending, masking, or similar means. The reviewer has experience with studies on posterior probability and would appreciate confirmation.

3. The reviewer finds the reported speed and runtime of the proposed method particularly perplexing. Given the numerous loops in the algorithm, it is unclear how the method achieves a runtime of only 1.2 seconds. As far as the reviewer is aware, most training-free image editing methods, even with fewer layers in the algorithmic structure, typically take tens of seconds. Could the authors clarify where this dramatic time saving is achieved? Beyond seeing a 1.2-second result, the reviewer would like a breakdown of time components comparing this method to other training-free methods.

4. The number of baseline methods in the comparative experiments is insufficient, with few comparisons to state-of-the-art training-based methods. This lack of comparison feels inequitable, as training-based methods have become well-established for image editing, showing robustness and speed (e.g., InstructP2P, SmartEdit, SEED-X-Edit). The reviewer questions what specific contributions are offered by the proposed training-free method.

5. The proposed method involves an overwhelming number of hyperparameters; there appear to be countless hyperparameters even within the algorithm section alone. The reviewer believes that a detailed hyperparameter sensitivity analysis is necessary to ensure that the model's robustness is not significantly affected by the choice of hyperparameters. From the reviewer's experience, many training-free methods require intricate hyperparameter tuning to achieve optimal results. Therefore, the reviewer requests that the authors provide the specific hyperparameters used for each image in the experimental section.

6. The related work section appears to reference too few relevant studies in the image editing field, which is rapidly evolving. The reviewer suggests expanding this section by referencing additional survey papers to strengthen the literature review.

Meta-review: This paper introduces PostEdit, a novel method for zero-shot image editing that leverages posterior sampling to achieve both efficiency and background preservation. The paper claims that PostEdit overcomes the limitations of existing inversion-based and inversion-free methods by incorporating a posterior scheme to guide the diffusion sampling process. The paper introduces a measurement term related to both initial features and Langevin dynamics to optimize the estimated image generated by the target prompt. The key finding is that PostEdit achieves state-of-the-art editing performance while preserving unedited regions with high efficiency.  

Strengths:
- Efficiency: PostEdit is both inversion- and training-free, requiring only 1.5 seconds and 18 GB of GPU memory for high-quality results.  
- Background Preservation: The method effectively preserves unedited regions of the image, addressing a common challenge in image editing.  
- Theoretical Foundation: The incorporation of posterior sampling and Langevin dynamics provides a solid mathematical framework for the approach.  
- Clarity and Depth: The paper is well-written and presents the methodology with clarity and theoretical depth.  

Weaknesses:
- Limited Editing Operations: The initial experiments primarily focused on replacing a few words in the prompt, potentially limiting the scope of editing operations.  
- Lack of Complex Image Reconstruction: The reconstruction results mainly involved images with single objects and simple layouts, lacking demonstrations with complex images containing multiple detailed objects.  
- Missing Baselines: The initial experimental comparisons lacked some state-of-the-art methods, such as SPD Inversion and Guide-and-Rescale.  
- Limited Ablation Study: The initial ablation study lacked quantitative results to fully support the analysis of individual components' effectiveness.  

Despite the identified weaknesses, the paper presents a novel and efficient approach to zero-shot image editing with nice theoretical formulation. The method addresses important challenges in the field, particularly in terms of efficiency and background preservation. The authors have also diligently addressed the reviewers' concerns during the rebuttal period by providing additional experiments, clarifications, and analyses. The overall consensus among the reviewers, coupled with the paper's strengths and the authors' responsiveness, supports the decision to accept.

Decision: Accept (Poster)