Rethinking Attentions in Zero-Shot Real Image Editing

Authors: Dinh-Khoi Vo, Thanh-Toan Do, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: stable diffusion, image editing, zero-shot algorithm, attention

Abstract:
Editing natural images using textual descriptions in text-to-image diffusion models remains a significant challenge, particularly in achieving consistent generation and handling complex, non-rigid objects. Existing methods often struggle to preserve textures and identity, require extensive fine-tuning, and exhibit limitations in editing specific spatial regions or objects while retaining background details. This paper proposes Context-Preserving Adaptive Manipulation (CPAM) -- a novel zero-shot method for complicated, non-rigid real image editing. Specifically, we propose a preservation adaptation module that adjusts self-attention mechanisms to effectively preserve and independently control the object and background. This ensures that the objects' shapes, textures, and identities are maintained while keeping the background undistorted during the editing process using the mask guidance technique. Additionally, we develop a localized extraction module to mitigate the interference with the non-desired modified regions during conditioning in cross-attention mechanisms. We also introduce various mask-guidance strategies to facilitate diverse image manipulation tasks in a simple manner. Extensive experiments on our newly constructed Image Manipulation BenchmArk (IMBA), a robust benchmark dataset specifically designed for real image editing, demonstrate that our proposed method is the preferred choice among human raters, outperforming existing state-of-the-art editing techniques.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729908641466

thread_type:
official_review

<thread_object>
creator: reviewer_3qqf

<content>
summary:
This paper proposes a Context-Preserving Adaptive Manipulation method for real image editing. Preservation adaptation module and localized extraction module are introduced for keeping objects maintained and mitigating interferences. Additionally, some mask-guidance strategies and a new benchmark are mentioned. The proposed method demonstrates better results in terms of context maintenance.

strengths:
1. The insight of the paper is good, and the implementation of image editing with background preservation is close to the actual application scenarios.
2. The visualized results show that the proposed method achieves the desired effect.
3. The structure of the paper is well organized.

weaknesses:
1. Although the problems attempted to be solved are interesting, the solutions lack innovation. 
2. The additional introduction of mask inputs leads to unfair comparisons with existing methods.
3. The proposed mask-guidance strategies are rule-based and not flexible enough to cope with diverse editing needs.
4. Inadequate visual comparison of ablation experiments.
5. The structure and presentation of figure 2 is not sufficiently clear.
6. English expression needs improvement: “retaining object is true”, “where s, l, S = 3, L = 8 denotes” and “element-wise dot product”.

questions:
1. How much does the accuracy of the input mask affect CPAM?
2. What will be the result of acquiring masks for CPAM inputs by means of adaptive perception or reference segmentation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565150287

thread_type:
official_review

<thread_object>
creator: reviewer_rup3

<content>
summary:
This paper proposes Context-Preserving Adaptive Manipulation (CPAM) for zero-shot real image editing. It includes three main components: 1) a preservation adaptation module that adjusts self-attention mechanisms to effectively preserve and independently control the object and background; 2) a localized extraction module that applies attention to the spatial pixels of the extracted object from the feature query to the target prompt, while the remaining pixels attend to a null text prompt; and 3) various masking strategies tailored to different editing needs.

strengths:
1. This paper is well-written and esay to read. 
2. The performance of this paper is promising, achieving state-of-the-art results across various metrics.
3. The insights on multi-text guided synthesis (Fig. 2 and Sec. 3.4) are interesting.

weaknesses:
1. Lacking of experimental evidence supporting the claim that "null text does not affect the output," as stated in Lines 206-211. 
2. The notation "t > T, l > L" in Fig. 3(b) is inconsistent with "s > S, l > L" in Eq. 2, and Fig. 3 requires further refinement.
3. The proposed method, CPAM, seems incremental because: 1) the main design, "Preservation Adaptation," is highly similar to "Mask-Guided Mutual Self-Attention" from the MasaCtrl paper; and 2) the masking strategy, which aggregates cross-attention maps across all steps and layers, also closely resembles that of MasaCtrl in Sec. 4.2. 
4. Lacking of ablation studies to validate the effectiveness of the proposed modules; only results under different classifier-free guidance scales are presented in Sec. A.1.

questions:
I look forward to your response to address my concerns outlined in the 'Weaknesses' section. I will adjust my score based on your reply and the ratings from the other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693246669

thread_type:
official_review

<thread_object>
creator: reviewer_rvma

<content>
summary:
The method proposes mask-guided image editing.

strengths:
Editing performance is improved from baselines.

weaknesses:
The method lacks novelty. The method of self-attention injection is already well-known method, and usage of mask-guidance is not a novel method. It seems the proposed method is combination of framework of Diffedit (automatic mask generation) and self-attention injection similar to Plug-and-play diffusion features. I think there is no new or technical idea which can further contribute to the generative AI field. 

Also, the quality of output edited images are not satisfactory. The paper proposes that the editing method can be applied to all kinds of editing, but it seems that the outputs can be obtained with using baseline methods with proper parameter control.

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713678341

thread_type:
official_review

<thread_object>
creator: reviewer_zhlc

<content>
summary:
- This paper presents Context-Preserving Adaptive Manipulation (CPAM) to address complex, non-rigid image editing using a tuning-free, zero-shot approach. 
- It utilizes both self-attention and cross-attention mechanisms in Stable Diffusion to enable intricate and text-guided image edits.
- It proposes the Preservation Adaptation Process to adjust self-attention layers to retain the identity, texture, and shape of the object being edited while the background remains unchanged.
- It also includes a Localized Extraction Module to avoid undesired changes to other image areas, and it enables the selective application of cross-attention.
- It proposes various strategies to control the editing scope based on the task, e.g. object removal, replacement, or background alteration, and it allows users to define which parts of the image are editable.
- The paper also introduces a new benchmark to evaluate real-image editing models and provides extensive qualitative and quantitative comparisons to show the effectiveness of the proposed method.

strengths:
- This paper offers useful advances for real-image editing by removing the need for fine-tuning and allowing flexible, zero-shot editing.
- This paper is organized well and gives clear explanations of each new component and how they work together to improve image editing.
- The author provides extensive experiments to show that the method works better than existing methods.
- This paper also introduces a new benchmark dataset for more comprehensive evaluation.

weaknesses:
- The paper proposes various new modules, but does not delve into ablation studies that isolate and analyze the impact of each component/module within the framework.
- The teaser figure shows unintended changes in the horse's view and texture, despite no prompt specifying these edits. This suggests some loss of subject-specific details. This also contradicts the authors' claim in Figure 14, where they state that without fine-tuning, the model cannot generate novel poses or views. It raises questions about the proposed method's control over subject attributes.
- Table 1 shows only marginal improvements in metrics that measure overall image quality and background. This makes the advantage unclear. Also, the paper should include some notes on each metric. For example, CLIP score could mean either prompt alignment (image-text) score or the subject-level image similarity (image-image) score.
- The dependency on high-quality masks can limit practicality in fully automated editing and in situations where generating precise masks is difficult.
- Some recent related works in the same domain:
	ProxEdit: Improving Tuning-Free Real Image Editing with Proximal Guidance
	EmuEdit: Precise Image Editing via Recognition and Generation Tasks

questions:
Could the authors provide an ablation study to isolate and analyze the impact of each component? Also, could the authors clarify the limitations of pose/view manipulation in the current framework?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732167592902

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewers: Innovation, Novelty, and Core Contributions

comment:
We appreciate the reviewers’ feedback and the opportunity to clarify the novelty and contributions of our proposed method. Below, we summarize the key aspects that highlight the novelty and practical impacts.

---

### **A. Core Contributions and Novelty**

**1. Innovative Framework**  
Our paper introduces a novel framework that includes:
- **Preservation Adaptation:** Independently manages object and background preservation across steps and layers to achieve precise edits without compromising non-target regions. Please kindly refer to **Figure 8 in Appendix A.2** for comparisons with MasaCtrl. 
- **Localized Extraction:** Pioneers multi-condition editing by leveraging both text and null text prompts to ensure adaptive control over diverse scenarios.
- **Mask-Guidance Strategy:** Handles rigid and non-rigid changes efficiently, enabling tasks like object removal, background replacement, while preserving background integrity, eliminating the need for **architectural modifications or altering the flow pipeline**.

Our framework offers precise, flexible, and user-friendly solutions for real-world image editing tasks.

---

### **B. Practical Significance of Mask-Guidance**

**1. Resolving Prompt Ambiguity:**  
Masks clarify users' intent in ambiguous prompts, such as "a photo of a tree in autumn," by allowing users to specify whether to modify the tree, the background, or specific regions.

**2. Precision for Local Edits:**  
Masks simplify isolating intricate regions (e.g., jewelry, textures) and was seamlessly integrated with user-drawn or automatically generated inputs.

**3. High Guidance Without Distortion:**  
Our approach supports high guidance scales while preserving background integrity and avoiding distortions, enabling accurate and realistic edits.

---

### **C. Scalability and Usability**

**1. Multi-Prompt and Multi-Region Editing:**  
Our method can be extended to complex scenarios where multiple prompts and masks are applied to distinct image regions. For example:
- To edit an image with the prompts **"wear sunglasses"** and **"wear an Apple Watch,"** users can assign corresponding masks to the eye and wrist regions. This functionality exceeds the capabilities of baseline methods limited to single-prompt and single-mask workflows.

**2. Balancing Automation and Manual Control:**  
Our method supports both automatic and user-drawn masks, ensuring adaptability to diverse user needs while maintaining precision and flexibility.

---

### **D. Valuable Insights from Self and Cross Attention**

Our exploration of self- and cross-attention mechanisms provides a foundation for advancing future image editing techniques:
- **Cross-Attention:**  
  By attending specific prompts to distinct parts of the latent noise, our CPAM ensures precise conditioning for edits. Null-text prompts maintain background integrity while focusing object modifications.
- **Self-Attention:**  
  Enables smooth transitions and cohesive edits by connecting pixels to themselves, ensuring semantic consistency and precise exclusion of regions (e.g., for object removal).

These insights emphasize the control and flexibility of our method, paving the way for future innovations in image editing.

---

### Conclusion

Our work introduces meaningful advancements in precision, flexibility, and adaptability for image editing, addressing key limitations of existing methods. We believe our innovative framework, practical mask-guidance strategy, and valuable insights into attention mechanisms demonstrate CPAM's novelty and impact in advancing real-world image editing techniques. We thank the reviewers for their constructive feedback and the opportunity to showcase our contributions.

We have revised the paper to address missing information, correct English expressions, resolve inconsistencies, and improve clarity and professionalism throughout the text.


We also provide ablation studies and demonstrate the impact of input mask accuracy on CPAM performance in [the anonymous rebuttal website](https://anonymous.4open.science/r/Anonymize-8672/Rebuttal.pdf).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents a Context-Preserving Adaptive Manipulation method for image editing task to identity is maintained and background is undistorted during the editing process. Two major raised concerns are limited contribution and lack of the important ablation study. It consistently receives four borderline reject from all reviewers. While part of concerns are addressed via rebuttal, the major concern about 
novelty still remains by at least explicitly mentioned by three reviewers. While there is some difference, AC agrees with all reviewers that the contribution of this work which bares a lot similarity with MasaCtrl is not big enough to meet the conference bar. Considering all the comments and discussions, a decision of reject is made. Authors are advised to continue improving the work based on reviews and resubmit elsewhere.