Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, an approach to controllable image editing with large-scale text-to-image diffusion models is proposed.
More specifically, an approach is presented that first analyzes the cross-attention (and self-attention) layers in the U-net backbone of state-of-the-art text-image diffusion models. It is then shown that these layers capture most of the structural information about the synthesized images.
This opens up a number of modification possibilities by directly modifying these attention maps. The paper demonstrates these modification possibilities using, among others, content modification and local semantic editing.
This overcomes a problem that arises when using "vanilla" prompt-based modifications, where small changes in the prompt can lead to very different synthesis outcomes.



strength_and_weaknesses: This is a strong empirical paper with convincing experimental results for editing with text-to-image diffusion models.
The finding that most of the object structure is stored in the cross-attention maps is interesting and has practical implications.
Several methods are presented for obtaining finer-grained, local control when editing text.
Finally, the manuscript is clearly written, easy to understand, and provides very good visual explanations.

A few questions that I had while reading the paper: 
How much control is lost by only applying the model on the 64x64 base model? Is this different in models using a diffusion model for upscaling or the decoder of an autoencoder? Does the "attention caching" lead to an improvement in sampling speed?



clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the presentation is very high. It was a real pleasure to read this work. As far as I know, this approach is novel and not only advances the processing capabilities of text-image diffusion models, but also opens up
a path to their interpretation.

Reproducibility:
Most of the experiments were performed (and presumably developed) on a closed-source system (Imagen) and are therefore not reproducible.
Although the results are certainly impressive, this is a clear drawback.
However, the work applies the method to publicly available systems and compares their performance, confirming that the approach works well for different types of methods.

summary_of_the_review: This is a very good empirical work with practical implications that greatly extends the applicability of modern text-to-image systems. The only major drawback is the fact that the method was mainly tested and evaluated on a closed-source system, which makes reproducibility difficult. However, since the method has also been validated on open systems, I am happy to recommend its acceptance for ICLR 2023.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. While large-scale language-image (LLI) models are collecting attention to generate an image from text, they have limitation to control by only text. The paper  propose to control the attention maps of the edited image by injecting the attention maps of the original image along the diffusion process. The paper shows diverse results by by editing the textual prompt only.
strength_and_weaknesses: Strength
1. The paper enables the user to control the generated image by text alone, by keeping the structure.
2. The proposed framework allows the user to control the extent.

Weakness
1. Failure cases are not shown for future work. There would be some failure cases or unsolved types of text edition. If the paper can discuss this aspect, the paper would be more appealing to readers.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. Editing the image by text alone while keeping the structure is very interesting and novel.
summary_of_the_review: The paper tries to solve an issue of controlling the image by text alone in image editing based on large-scale language-image models. By analyzing the cross-attention layers, the paper nicely shows a way to control the image by text alone. Diverse results show the effectiveness.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new method to only use text to achieve local image editing based on recent text-to-image models. More specifically, they propose to use cross-attention to connect each word and image regions and get rid of user-provided masks. Many interesting applications have been proposed and the results are very promising.
strength_and_weaknesses: Strengths
1. The proposed task of only using text for local image editing is very interesting and of good practical values.
2. The authors proposed to use cross attention to connect word and image regions, which is novel and reasonable. Although attention maps have been widely used in other tasks, I think it is quite appropriate here.
3. Many interesting applications have been provided. All results are promising and clearly show this method's advantages.
4. The paper writing is clear and easy to follow.

Weaknesses
1. Maybe some ablation study would be better. 
For example, does the proposed method affected by different noise sampler? 

2. For those tasks that only local edits are needed, is it possible to only process local area and accelerate the process?
clarity,_quality,_novelty_and_reproducibility: I think the proposed method is technically sound and the results are very promising. They addressed a practical issue and solved using novel and reasonable techniques. The authors also provided various interesting editing applications which help evaluate the method's value.

The paper writing is clear and figures are informative. The quality and clarity are satisfying.
summary_of_the_review: This paper proposes a new method to only use text to achieve local image editing based on recent text-to-image models. 
The method is novel and technically sound. And the author provided various new applications which further prove its effectiveness.

I think this method can also inspire many interesting future work in this area.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a text-driven image editing technique to refine images generated from large-scale text-to-image diffusion models. The paper relies on a crucial observation: Just modifying a text prompt when calling the generator results in structurally and compositionally very different images, even if the text prompt is modified only slightly and even if the same random seed is used. However, we can use the attention maps of the cross attention layers (that attend to the text tokens) corresponding to the initial image, and inject them into the generation process when generating a new image with a modified prompt. Now the new image will correspond structurally and compositionally to the original one, while also reflecting the new text prompt. This allows the authors to demonstrate rich text-driven editing capabilities. The method can be combined with masking schemes to localize the edits and re-weighting techniques to give more or less weight to certain words in the prompt. The approach is compared to several baselines and shows favourable results.
strength_and_weaknesses: **Strengths:**

- The proposed idea, modifying and re-using the attention maps of a diffusion model from one image, when generating a new one, is simple, yet elegant and novel. The approach makes intuitively sense and works well in practice.
- Generally, the paper is well-written, well-motivated and easy to follow (also see below). The work is also appropriately put into the broader context in the literature.
- Experimentally, the editing results are quite strong and the original structure and composition of the edited images is usually indeed well preserved when a new text prompt is used for editing. The fact that this is purely text-driven and yet allows fairly fine control is appealing.

**Weaknesses:**

- The work is only partially reproducible. The Imagen model is not available to the public and cannot be re-trained easily. The results on Stable Diffusion and the Latent Diffusion Model can be worse in some cases (see Appendix D.1).
- The approach comes with some hyperparameters. For instance, to achieve best results the attention map injection should only happen up to a certain time stamp during the iterative synthesis process. It seems like this time stamp would need to be tuned for each example separately, which can be costly, since running the diffusion model synthesis process repeatedly is expensive. This may not be ideal for interactive, real-time applications.
- Editing given real images is difficult, because this first requires an inversion process as well as finding an appropriate text prompt that could have generate the given image (this is discussed in the paper).
- As also pointed out by the authors themselves, other minor weaknesses include that the attention maps that are modified are in the low resolution layers of the U-Net, which prevents very fine-grained editing. Furthermore, editing corresponding to large-scale object movement is not possible with the proposed technique.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is clearly written and was easy to follow and read for me. I didn't spot any typos or other mistakes.

**Quality:** I think the overall quality of the work is high. State-of-the-art models are used. Appropriate experiments are run, including a user study. All evaluations seem correct. Appropriate baselines have been used in comparisons. 

**Novelty:** The approach is novel, to the best of my knowledge. The finding that the attention maps in the cross attention layers capture the structure of the text-image alignment and can even be re-used in new synthesis processes is significant and could find re-use in other works.

**Reproducibility:** The reproducibility of the work is somewhat limited. While the work also uses the publicly available Latent Diffusion Model and Stable Diffusion, the main model used in most experiments is Imagen, which is not publicly available and cannot easily be re-trained.
summary_of_the_review: In summary, I think this is a strong paper, which I recommend for acceptance. The methodology is based on a smart idea and novel, and the results are appealing. The paper is of high-quality throughout and well written and clearly presented. The paper has a couple of small weaknesses and the authors have mostly been transparent about that, but I do not think there are any major flaws. I believe the method could find practical use in the generative modeling community, for instance for producing generative art with even finer control.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The paper proposes a prompt-editing method for text-to-image diffusion models that utilize cross-attention layers in their design. It studies the following prompt-editing setting. There are a source prompt and a target prompt where the target prompt is derived from the source prompt. Only one visual concept differs in these two prompts. The goal here is to keep most of the source prompt-driven text-to-image output the same and only modify the part of the image where the source and target prompt disagree. The observation made in the paper is that such a task can be achieved by manipulating the attention maps for the cross-attention layers. The paper gives a clear exposition of why this can be achieved by manipulating the attention maps. The presented algorithm is easy to follow. The results are impressive.

The paper receives 4 reviews. All of them consider the paper above the bar, citing various strengths such as the paper being well-motivated, the algorithm being well-designed and easy to follow, and the application being useful and interesting. The meta-reviewer agrees with the assessment and would like to recommend acceptance of the paper.
justification_for_why_not_higher_score: This is a nice application paper. The presented idea is only applied to a subset of DL models for a specific application. The impact it creates will be limited to a small sub-field.
justification_for_why_not_lower_score: It is clearly a good paper with good motivation, presentation, algorithm design, and results.
Decision: Accept: notable-top-25%