PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editing

Authors: Feng Tian, Yixuan Li, Yichao Yan, Shanyan Guan, Yanhao Ge, Xiaokang Yang
Primary area: generative models
Keywords: Measurements, Posterior Sampling, Image Editing, Inversion-free, Trianing-free.

Abstract:
In the field of image editing, three core challenges persist: controllability, background preservation, and efficiency. Inversion-based methods rely on time-consuming optimization to preserve the features of the initial images, which results in low efficiency due to the requirement for extensive network inference. Conversely, inversion-free methods lack theoretical support for background similarity, as they circumvent the issue of maintaining initial features to achieve efficiency. As a consequence, none of these methods can achieve both high efficiency and background consistency. To tackle the challenges and the aforementioned disadvantages, we introduce PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt. Extensive experimental results indicate that the proposed PostEdit achieves state-of-the-art editing performance while accurately preserving unedited regions. Furthermore, the method is both inversion- and training-free, necessitating approximately 1.5 seconds and 18 GB of GPU memory to generate high-quality results.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730519860657

thread_type:
official_review

<thread_object>
creator: reviewer_cdse

<content>
summary:
This paper introduces PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt.

strengths:
This paper introduces PostEdit, a method that incorporates a posterior scheme to govern the diffusion sampling process. Specifically, a corresponding measurement term related to both the initial features and Langevin dynamics is introduced to optimize the estimated image generated by the given target prompt.

weaknesses:
1. The reviewer encountered several points in the algorithm that were difficult to understand, raising the possibility of inaccuracies. The reviewer requests that the authors provide clear responses to the following questions: (1) In line 4, does \(z_0\) bear no relation to the original image, given that \(z_N\) is derived from pure noise? (2) Can lines 6 to 9 be interpreted as a re-generation of \(z_0\)? (3) Is the alignment with the original image (i.e., \(z_{in}\)) genuinely implemented only in lines 12 to 14? (4) In line 15, the reviewer finds the superscript \(N\) on \(z^{(N)}\) unusual, as it would conventionally be \(m\) in this context. Additionally, the reviewer is puzzled by the disappearance of \(z_{\tau_{i-1}}\) in the next loop iteration. These uncertainties make the methodology difficult to follow, and the reviewer suggests clarifying these points for future readers.

2. Regarding the essence of the proposed method, the reviewer interprets it as generating an image that aligns with the target prompt, followed by posterior probability correction techniques that reintroduce information from the original image via weighted blending, masking, or similar means. The reviewer has experience with studies on posterior probability and would appreciate confirmation.

3. The reviewer finds the reported speed and runtime of the proposed method particularly perplexing. Given the numerous loops in the algorithm, it is unclear how the method achieves a runtime of only 1.2 seconds. As far as the reviewer is aware, most training-free image editing methods, even with fewer layers in the algorithmic structure, typically take tens of seconds. Could the authors clarify where this dramatic time saving is achieved? Beyond seeing a 1.2-second result, the reviewer would like a breakdown of time components comparing this method to other training-free methods.

4. The number of baseline methods in the comparative experiments is insufficient, with few comparisons to state-of-the-art training-based methods. This lack of comparison feels inequitable, as training-based methods have become well-established for image editing, showing robustness and speed (e.g., InstructP2P, SmartEdit, SEED-X-Edit). The reviewer questions what specific contributions are offered by the proposed training-free method.

5. The proposed method involves an overwhelming number of hyperparameters; there appear to be countless hyperparameters even within the algorithm section alone. The reviewer believes that a detailed hyperparameter sensitivity analysis is necessary to ensure that the model's robustness is not significantly affected by the choice of hyperparameters. From the reviewer's experience, many training-free methods require intricate hyperparameter tuning to achieve optimal results. Therefore, the reviewer requests that the authors provide the specific hyperparameters used for each image in the experimental section.

6. The related work section appears to reference too few relevant studies in the image editing field, which is rapidly evolving. The reviewer suggests expanding this section by referencing additional survey papers to strengthen the literature review.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613354339

thread_type:
official_review

<thread_object>
creator: reviewer_98bk

<content>
summary:
The paper introduces "PostEdit," a novel inversion- and training-free approach leveraging posterior sampling for efficient zero-shot image editing. The method addresses significant challenges in image editing, including controllability, background preservation, and efficiency. The overall results indicate promising performance, particularly in maintaining high-quality background similarity and efficiency in execution time. Nevertheless, there are a few methodological and experimental aspects that require further clarification and expansion.

strengths:
1. The proposed method's efficiency in terms of GPU memory and inference time is noteworthy. Achieving high-quality results in approximately 1.5 seconds and with only 18 GB of GPU memory is an impressive step forward for zero-shot image editing.
2. The authors provide strong mathematical support for their approach, with the incorporation of Langevin dynamics and a posterior measurement term to optimize the estimated image. This theoretically addresses the issue of error accumulation seen in existing approaches.
3. The results—both qualitative and quantitative—demonstrate the approach's effectiveness, showing its superiority compared to some existing methods.

weaknesses:
1. Methodological Clarifications:

a. Figure 2: Step 3 in Figure 2 is challenging to understand and requires more explanation regarding the optimization process. It would be beneficial to clearly explain how Langevin dynamics and the measurement term are incorporated in this step to improve the optimization.

b. The explanation of Symbols such as $\mathcal{A}$ are missing: In Equation (16), the symbol $\mathcal{A}$ is not clearly defined or explained. Adding a clear definition for this symbol would enhance the reader's understanding. Eq.16 needs more explanations for specific symbols such as m, h, and etc.

c. In lines 321-322, the authors mention the mask, however, how to obtain the mask, it seems that the paper without further explanations.

2. Experimental Clarifications:

a. Baselines: The experimental comparisons do not include state-of-the-art methods such as SPD Inversion [1], and Guide-and-Rescale [2]. Including these baselines is crucial to substantiate the claims of superiority. 

b. Ablation Study: The ablation study lacks quantitative results to further support the analysis. Numerical data on the effectiveness of each component (e.g., posterior sampling vs. inversion process) would add robustness to the conclusions.

c. User Study: Given that image editing quality can be somewhat subjective, a user study would be helpful to establish the practical effectiveness of the proposed method more convincingly.

d. Complex Images: All the reconstruction results shown in the paper involve images with a single object and simple layouts. However, some inversion methods struggle with reconstructing images containing multiple detailed objects. Could you provide reconstruction results for such complex images to strengthen your claims about the method's versatility?


References for Missing Baselines: 
[1] Li R B, Li R H, Guo S, Zhang L. Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models. European Conference on Computer Vision. 2024.

[2] Titov V, Khalmatova M, Ivanova A, et al. Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing. European Conference on Computer Vision. 2024.

questions:
Please refer to the "Weaknesses" section above for my questions and concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636640576

thread_type:
official_review

<thread_object>
creator: reviewer_psjc

<content>
summary:
This paper introduces a diffusion-based method designed for image reconstruction and editing, which operates within an encoded latent space. The proposed Langevin update rule for the latent vector at time $t+1$ consists of three components: a convex combination of the initial clean latent vector and the latent vector at time $t$, a negative gradient sum that guides the vector toward the initial noisy latent vector, and the measurement, and added Gaussian noise. 

The authors evaluate the proposed method through image reconstruction and editing experiments, providing qualitative and quantitative comparisons with state-of-the-art approaches. In editing experiments, the method outperforms the competitors in CLIP Similarity and achieves competitive results in other metrics while also being one of the fastest methods.

strengths:
The proposed algorithm is both inversion-free and training-free, contributing to its fast performance. The manuscript is well-written and clear. In editing experiments, the algorithm outperforms state-of-the-art approaches in CLIP similarity.

weaknesses:
In image restoration experiments, quantitative evaluation is lacking, as the authors provide only qualitative comparisons with competing methods. Furthermore, the authors do not provide evidence that the reconstruction results produced by their method are consistent with the input measurements. Specifically, it would be helpful to see whether applying the forward measurement operator to the algorithms' output yields results that closely approximate the original measurements.

questions:
- In equation 16, the gradient symbol is missing before the large brackets. The same issue occurs in line 13 of Algorithm 1 and row 710 in Algorithm 2.

- For a more comprehensive background, it may be beneficial to include a citation to [1], which introduced a method for solving linear image restoration problems in cases with noisy observations.

[1] Kawar, Bahjat, Gregory Vaksman, and Michael Elad. "SNIPS: Solving noisy inverse problems stochastically." Advances in Neural Information Processing Systems 34 (2021): 21757-21769.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727114026

thread_type:
official_review

<thread_object>
creator: reviewer_6fqq

<content>
summary:
This paper tackles the issues caused by the unconditional term in Classifier-Free Guidance (CFG) by integrating the theory of posterior sampling to improve reconstruction quality for image editing. By reducing the reliance on repeated network inference, the proposed method, PostEdit, achieves fast and accurate performance while successfully maintaining background consistency. The results seem promising.

strengths:
1. By extending the theory of posterior sampling to text-guided image editing tasks, the proposed method, PostEdit, eliminates the need for both inversion and training. 

2. PostEdit is one of the fastest zero-shot image editing approaches, achieving execution times of under 2 seconds on an A100 GPU.

weaknesses:
While the paper presents an interesting approach, some details and experimental results are not sufficiently comprehensive. Key aspects of the methodology are not fully elaborated, and additional experiments would be beneficial to further validate the claims. 

1. The subfigure "Our Posterior Sampling Process" in Fig. 1 is difficult to understand. It is unclear what exactly it is meant to represent. Additionally, how does it highlight the advantages of the proposed algorithm?

2. The authors mention that the proposed algorithm runs in less than 2 seconds, but it is not clear how many denoising steps are actually performed. In Algorithm 1, the values of N and n are not specified. These details are crucial because if the number of denoising steps is high, it could naturally lead to slower execution. It would also be beneficial to visualize some intermediate results for better clarity.

3. Regarding the editing process, how is the mask generated and provided? There is a lack of details on this aspect.

4. The current editing experiments only involve replacing a few words in the prompt, which seems quite limited. Does the method support more extensive editing operations? More explanation and additional experiments would be appreciated.

questions:
Please see the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732283385261

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of The Contents of Our Reply

comment:
**We sincerely thank all reviewers for their time and effort in reviewing our paper. We are thrilled to see that the reviewers recognized the novelty of our contributions (Reviewers 6fQq, Psjc, 98Bk, CdSe) and appreciated the paper's clarity and theoretical depth (Reviewers Psjc, 98Bk). We are looking forward to your further discussion and questions.** 

We also sincerely appreciate the reviewers for their professional, specific, and constructive comments and concerns. To address these, we have made modifications and provided detailed explanations in the attached PDF file for further clarification. Below, we summarize the key contents included in the attached file:

- **Figure 1 (to Reviewer 6fQq)** is revised for better comprehension of the presented three categories.
- **Algorithm 1 and 2 (to Reviewer Psjc)** are revised according to the reviewer's advice.
- **The operator $\mathcal{A}$ (to Reviewer 98Bk)** shown in Eq.(16) is illustrated and the content is following Eq.(16).
- **Citation [1] (to Reviewer Psjc)** is added in the introduction and method sections.
- **Figure 3 and Table 1 (to Reviewer 98Bk, CdSe)** are updated.
- **Appendix A. 3, Appendix A. 5, Appendix A. 6, Appendix A. 7, Appendix A. 8, Appendix A. 9, Appendix A. 10 and Appendix A. 12 (to Reviewer 6fQq, Psjc, 98Bk, CdSe)** have been added to show required experimental results.
- **A 'Related Work' section (to Reviewer CdSe)** is added in Appendix A.1.

Last, the presentation of the full text has been optimized to enhance clarity and improve understanding.

[1] Kawar, Bahjat, Gregory Vaksman, and Michael Elad. "SNIPS: Solving noisy inverse problems stochastically." Advances in Neural Information Processing Systems 34 (2021): 21757-21769.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces PostEdit, a novel method for zero-shot image editing that leverages posterior sampling to achieve both efficiency and background preservation. The paper claims that PostEdit overcomes the limitations of existing inversion-based and inversion-free methods by incorporating a posterior scheme to guide the diffusion sampling process. The paper introduces a measurement term related to both initial features and Langevin dynamics to optimize the estimated image generated by the target prompt. The key finding is that PostEdit achieves state-of-the-art editing performance while preserving unedited regions with high efficiency.  

Strengths:
- Efficiency: PostEdit is both inversion- and training-free, requiring only 1.5 seconds and 18 GB of GPU memory for high-quality results.  
- Background Preservation: The method effectively preserves unedited regions of the image, addressing a common challenge in image editing.  
- Theoretical Foundation: The incorporation of posterior sampling and Langevin dynamics provides a solid mathematical framework for the approach.  
- Clarity and Depth: The paper is well-written and presents the methodology with clarity and theoretical depth.  

Weaknesses:
- Limited Editing Operations: The initial experiments primarily focused on replacing a few words in the prompt, potentially limiting the scope of editing operations.  
- Lack of Complex Image Reconstruction: The reconstruction results mainly involved images with single objects and simple layouts, lacking demonstrations with complex images containing multiple detailed objects.  
- Missing Baselines: The initial experimental comparisons lacked some state-of-the-art methods, such as SPD Inversion and Guide-and-Rescale.  
- Limited Ablation Study: The initial ablation study lacked quantitative results to fully support the analysis of individual components' effectiveness.  

Despite the identified weaknesses, the paper presents a novel and efficient approach to zero-shot image editing with nice theoretical formulation. The method addresses important challenges in the field, particularly in terms of efficiency and background preservation. The authors have also diligently addressed the reviewers' concerns during the rebuttal period by providing additional experiments, clarifications, and analyses. The overall consensus among the reviewers, coupled with the paper's strengths and the authors' responsiveness, supports the decision to accept.