Prompt-to-Prompt Image Editing with Cross-Attention Control
Authors: Amir Hertz
Ron Mokady
Jay Tenenbaum
Kfir Aberman
Yael Pritch
Daniel Cohen-Or
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Image generation
Image editing
Diffusion models
Attention layer
Computer vision
Machine learning

Abstract: Recent large-scale text-driven synthesis diffusion models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Therefore, it is only natural to build upon these synthesis models to provide text-driven image editing capabilities. However, Editing is challenging for these generative models, since an innate property of an editing technique is to preserve some content from the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. We analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we propose to control the attention maps along the diffusion process. Our approach enables us to monitor the synthesis process by editing the textual prompt only, paving the way to a myriad of caption-based editing applications such as localized editing by replacing a word, global editing by adding a specification, and even controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts with different text-to-image models, demonstrating high-quality synthesis and fidelity to the edited prompts.

Published as a conference paper at ICLR 2023
PROMPT-TO-PROMPT IMAGE EDITING WITH CROSS-ATTENTION CONTROL
Amir Herty 1.2, Ron Mokady 122 Jay Tenenbaum Kfir Aberman', Yael Pritch', and Daniel Cohen-O(f
1 "Google Research
The Blavatnik School of Computer Science, Tel Aviv University
ABSTRACT
Recent large-scale text-driven synthesis diffusion models have attracted much at- tention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Therefore, it is only natural to build upon these synthesis models to provide text-driven image editing capabilities. However, Edit- ing is challenging for these generative models, since an innate property of an edit- ing technique is to preserve some content from the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-ii art methods mitigate this by requir- ing the users to provide spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-to-promp editing framework, where the edits are controlled by text only. We analyze text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we pro- pose to control the attention maps of the edited image by injecting the attention maps of the original image along the diffusion process. Our approach enables us monitor the synthesis process by editing the textual prompt only, paving the editing applications such as localized editing by
way to a myriad of caption- replacing word, global editing by adding a specification, and even controlling the extent to which word is reflected in the image. We present our results over diverse images and prompts with different text-to-image models, demonstrating high-quality synthesis and fidelity to the edited prompts.
INTRODUCTION
Recently, large-scale language-image (LLI) models, such as Imagen (Saharia et al. 2022b), DALL-E (Ramesh et al. 2022) and Parti (Yu et al. (022). have shown phenomenal generative semantic and compositional power, and gained unprecedented attention from the research commu- nity and the public eye. These LLI models are trained extremely large language-image datasets and use state-of-the-art image generative models including auto-regressive and diffusion models. However, these models do not provide simple editing means, and generally lack control over spe- cific semantic regions of a given image. In particular, even the slightest change in the textual prompt may lead completely different output image. To circumvent this, LLI-based methods (Nichol et al.) 2021 Avrahami et al., 2022a Ramesh et al. 2022) require the user to explicitly mask a inpainted, and drive the edited image change the masked area only,
part of the image while matching the background of the original image. This approach has provided appealing re- sults, however, the masking procedure is cumbersome, hampering quick and intuitive text-driven editing. Moreover, masking the image content removes important structural information, which is completely ignored in the inpainting process. Therefore, some capabilities are out of the inpainting scope, such modifying the texture of specific object.
In this paper, we introduce an intuitive and powerful textual editing method to semantically edit images in pre-trained text-conditioned diffusion models via Prompt-to-Prompp manipulations. To do so, we dive deep into the cross-attention layers and explore their semantic strength as a handle to control the generated image. Specifically, we consider the internal cross-attention maps, which are
Perrformdd this work while working Google.

---

Published as a conference paper at ICLR 2023
"The boulevards are crowded today." 743444
"Photo of riding Bleucle." CBr
"Landscape with a house near a river and rainbow the baerground!
11:51 "My fluffy bunny doll."
"a cake with decorations."
"Cildrtn drewnngg of castle next river."
Jely Veaus
Figure 1: Prompt-toPrrome editing capabilities. Our method paves the way for myriad of caption-based editing operations: tuning the level of influence of an adjective word (bottom-left), making local modification in the image by replacing adding word chottom-middlee fying global modification (bottom-right)
speci-
nigh-dimensional tensors that bind pixels and tokens extracted from the prompt text. We find that these maps contain rich semantic relations which critically affect the generated image.
Our key idea is that process, controlling which pixels attend to which tokens of the prompt text during which diffusion steps. To apply our approach to various creative editing applications, control the cross-attention maps through a simple and semantic interface (see fig. [ The first is to change single token's value in the prompt (e.g., "dog" to "cat"), while fixing the cross-attention maps, to preserve the scene composition. The second is adding new words to the prompt and freezing the attention on previous tokens while allowing new attention flow to the new tokens. This enables us to perform global editing or modify specific object. The third is to amplify or attenuate the semantic effect of word in the generated image. Furthermore, we demonstrate how to use these attention maps obtain a local editing effect that accurately preserves the background.
edit images by injecting the tross-attention maps during the diffusion
show several methods to
Our approach constitutes an intuitive image editing interface through editing only the textual prompt, therefore called Prompt-to-PP Prooppp This method enables various editing tasks, which are chal- lenging otherwise, and does not require model training, fine-tuning, extra data, or optimization. Throughout analysis, discover even more control over the generation process, recognizing trade-off between the fidelity to the edited prompt and the source image. We also demonstrate that method operates with different text-to-image to-ieae models for the public models upon acceptance. Finally, our method even applies to real images by using an existing inversion technique. Our experiments show that our method enables intuitive text-based editing over diverse images that current methods struggle with.
backbone and we will publish code
2 RELATED WORK
Image editing is one of the most fundamental tasks in computer graphics, encompassing the process of modifying an input image through the use of an auxiliary input, such as a label, mask, or refer- ence image. A specifically intuitive way to edit an image is through textual prompts provided by the user. Recently, text-driven image manipulation has achieved sigmificant progress using GANs (Goodfellow et af. 2014 Brock et al., 2018 Karras et al. 2019). which are known for their high- quality generation, tandem with CLIP (Radford et al. 20011) which consists of semantically rich joint image-text representation, trained over millions of text-image pairs. Seminal works (Patashnik) et al. 2021 Gal et al. 2021 Xia et al. 2021a) which combined these components were revolution- ary, since they did not require extra manual labor, and produced realistic manipulations using text only. For instance, [au et al. (2021) further demonstrated how to masks to restrict the text-based editing to a specific region. However, while GAN- editing approaches succeed on curated data, e.g., human faces, they struggle over large and diverse datasets (Mokady et al. 2022).
To obtain more expressive generation capabilities, Crowson et al. (2022) use VQ-GAN (Esse et al. 2021b), trained over diverse data, as backbone. Other works (Avrahami et al. 2022b Kim al.
2022) exploit the recent Diffusion models (Ho et al. 2020 SSng & Ermon 2019 Ho el al. 2020 Song et al., 2020; Rombach et al. 2021 [H al., 2022 Saharia et | 202 2022a which achieve -fate-of-the-art generation quality over diverse datasets, often surpassing GANs Ns (Dhariwal & Nichol,
2021). Kim al. (2022) show how to perform global changes, whereas Avrahami al. (2022b) suc-
2

---

Published as a conference paper at ICLR 2023
fixed attention maps and random seed
ooo
Temoo
Pppple Gike
Thoola rake
Tost fixed randon seed
"paad
"lagn cale."
"mnnet cake."
'letion cake.
ayple caee"
"doodtne cake.
"boot
"y4Ra cake.
cake
"maitsee cake."
"brick
Figure 2: Content modification through attention injection. We start from an original image generated from the prompt "lemon cake" (top left). and modify the text prompt to a variety of other cakes. On the top row, we inject the attention weights of the original image during the difflssion process. On the bottom, we only use the same random seeds the original image, without injecting attention. The latter leads to a completely new structure that is hardly related to the original.
cessfully perform local manipulations using user-provided masks for guidance. While most works that require only text (i.e., no masks) are limited to global editing (Crowson et al., 2022, K Koon &
Ye 2021. Bar- Tal et (2022) proposed text-based localized editing technique without using any mask. showing impressive results. Yet, their techmiques mainly allow changing textures, but not modifying complex structures, such as changing bicycle car. Moreover, unlike our method,
their approach requires training network for each input.
Numerous works (Ding al.) [00T]] Hinz et al. 2020 Tao al. 2020 Li al. 2019 Ramesh al,,
2021 Zhang et al. 2018b Crowson al.) [2022] Gafni al. 2022 Rombach al. 2021 advanced the generation of images conditioned on plain text, known as text-to-image synthesis. But only re- cently these were followed by several large-scale text-image models, such as Imagen (Saharia al,, 2022b), DALL-E2 (Ramesh et al., 2022). ) and Parti YYu et al. 2022), demonstrating unprecedented semantic generation. However, these models do not provide control over generated image, specif- ically using text guidance only. Changing single word in the original prompt associated with the image often leads completely different outcome. For instance, adding the adjective "white" to "dog" often changes the dog's 's shape. To overcome this, several works (Nichol et al. [222 Avrahami al. 2022a) assume that the user provides mask to restrict the edited region.
Unlike previous works, our method requires textual input only, by using the spatial information from the internal layers of the generative model itself. This offers the user a much more intuitive editing experience of modifying local or global details by merely modifying the text prompt.
METHOD
Let I be an image that was generated by a text-guided diffusion model using the text prompt P and a random seed s. Our goal is to edit I, using only the guidance of an edited prompt Pt, in order get an edited image I" that maintains the content and structure of the original image but corresponds to the edited prompt. For example, consider an image generated from the prompt "my new bicycle", and assume that the user wants to edit the color of the bicycle or replace it with scooter while preserving the appearance and structure of the original image. An intuitive interface for the user to directly change the text prompt by further describing the appearance of the bike, replacing it with another word, respectively. As opposed to previous works, we wish to avoid relying on any user-defined mask to assist or signify where the edit should occur. simple, but unsuccessful attempt to fix the internal randomness and regenerate using the edited text prompt. Unfortunately, as fig. [ shows, this results in a completely different structure and composition.
Our key observation is that the structure and appearance of the generated image depend not only on the random seed, but also on the interaction between the pixels to the text embedding through the diffusion process. By modifying the pixel-to-text interaction that occurs in cross-atlention layers, image editing capabilities. More specifically, injecting the cross-
we provide Prompt-to-Prome attention maps of the input image I enables to preserve the original composition and structure. In Section we review how cross attention is used, and in Section 3.2
we describe how to exploit the
cross-attention for editing. Self-attention is discussed in section
33 For background on diffusion
models, refer to appendix A
3

---

Published as a conference paper at ICLR 2023
Attention maps
Pixel features
Pixel Queries
Tokens Keys (from Prompt)
Tokens Values (from Prompt)
Output
(2L)
K
M2
(#1)
Text to Image Cross Attention Cross Attenetion Control
New weighting
M
M2
Mi
M
M,
Word Swap
Prompt Refinement
Attention Re- Re-weighting
Figure 3: Method overview. Top: visual and textual embedding are fused using cross-attention layers that produce attention maps for each textual token. Bottom: we control the spatial layout and geometry of the generated image using the attention maps of source image. This enables various editing lasks through editing the textual prompt only. When swapping word in the prompt, we inject the source image maps M,, overriding the target maps Mc. In the case of adding refinement phrase, we inject only the maps that correspond to the unchanged part of the prompt. To amplify or attenuate the semantic effect of a word, we re-weight the corresponding altention map.
3.1 GROSS-ATTENTEE
IN TEXT-CONDITIONED DIFFUSION MODELS
In this section, we refer to the Imagen (Saharia et al., 2022b) text-guided synthesis model as our backbone, although our method is not limited to a specific model, and results with Latent Diffusion and Stable Diffusion (Rombach et al., 2021) are presented in section 4.1 and appendix C All three models condition on the text prompt in the noise prediction of each diffusion step throughout cross-attention layers. For further details about the attention layers within each model, please see appendix A.2 Since the composition and geometry are mostly determined the 64 x 64 resolution, we only adapt the text-to-ing tonttto-mmae diffusion model, using the super-resolution process as is. Recall that each diffusion step consists of predicting the noise from noisy image 2 and text embedding (P) using U-shaped network (Ronneberger et al., 2015). At the final step, this process yields the generated image = =0- Most importantly, the interaction between the two modalities occurs during the noise prediction, where the embeddings of the visual and textual features are fused using cross-attention layers that produce spatial attention maps for each textual token. More formally, as illustrated in fig. (top), the deep spatial features of the noisy image ((2)) are projected query matrix Q = (2(((())) and the textual embedding is projected to a key matrix value matrix V (1))) via learned linear projections la, kx, Ey. Attention maps then
Ex (v)))) and
QKT
M Soofmmx
Vd
where the cell My defines the weight of the value of the j-th token the pixel i, and dl is the latent projection dimension of the keys and queries. Finally, the cross-attention output is defined to be á (z)) = MV, which is then used to update the spatial features o(2)).
Intuitively, the cross-attention output MV is weighted average of the values V where the weights are the attention maps M, which are correlated to the similarity between Q and K. In practice, to increase their expressiveness, multi-head attention (Vaswani et al., 2017) is used in parallel, and then the results are concatenated and passed through a learned linear layer to get the final output.
3.2 CONTROLLING THE CROSS-ATTENTION
We return to our key observation - the spatial layout and geometry of the generated image depend on the cross-attention maps. The interaction between pixels and text illustrated in fig. 4 where the average attention maps are plotted. As be seen, pixels are more attracted to the words that describe them, e.g., pixels of the bear are correlated with the word "bear". Note that averaging is done for visualization purposes, and attention maps kept separate for each head. Interestingly, we can see that the structure is already determined in the early steps of the diffusion process.
Since the attention reflects the overall composition, obtained from the generation with the original prompt P, into a second generation with the modified prompt P This allows the synthesis of edited image I that is not only manipulated according
inject the attention maps M that were
4

---

Published as a conference paper at ICLR 2023
Syothesized image
furry
betr
watches
a
Average cross-attention maps across all timestamps Cross-attentie attention maps individual timestamps
bear
Figure 4: Cross-attention maps of a text-conditioned diffusion image generation. Top: average attention masks for each word in the prompt which was used to synthesize the left image. Bottom: attention maps with respect to the word "bear" from different diffusion steps, ranging from the first step T = 256 to the last step t in equal intervals.
to the edited prompt, but also preserves the structure of the input image I. This is specific instance of a broader set of attention-base manipulations leading to different types of intuitive editing. We, therefore, start by proposing general framework, followed by the details of the specific operations.
Let (2, s) be the computation of a single step of the diffusion process, which out- puts the noisy image Zt-1, and the attention map M, (omitted if not used). We denote by (M, P.t, {M
M) the diffusion step where additional given map M, but keep the values V from the supplied prompt. We also denote by Mf the produced attention map using the edited prompt P Lastly, we define Edit(Mr M,,, () to be general edit function, receiving as input the l'th attention maps of the original and edited images.
override the attention map A with an
Our general algorithm for controlled generation consists of performing the iterative diffusion process for both prompts simultaneously. where an attention- based manipulation is applied in each step according to the desired editing task. We fix the internal randomness since even for the same prompt, two random seeds produce drastically different outputs. We also define local editing scheme in subsequent paragraph. Formally, our general algorithm for editing the image I, which is generated by prompt P and seed s, is define::
Algorithm 1: Prompt-to to- Po-Prompt image editing
Input: source prompt P, target prompt p", and a random seed s.
Optional for local editing: w and w", words in P and Pt, specifying the editing region.
Output: source image Tsre and an edited image ddst- 2T ~ (0,, I) unit Gaussian random variable with random seed
2T
2T;
for
...., T,T 1....,, do
2;-1, M2 -- (M P, P,t,8);
M2 + DM P ,t,s); M, + (M, Mí;t);
2 + -at P st Mi):
10
if local then
- B B(M BMM (.w-): 2 - (1
12
13
2t-1 2
14
end
end
16
Return (20,2
For editing real images, see section Also, note that we can skip the forward call in line 8 by applying the edit function inside the diffusion forward function. Moreover, diffusion step can be applied on both z,- and z in the same batch (i.e., in parallel). We now turn to address local editing followed by specific editing operations, filling the missing definition the Edit(Me Mf t) function. An overview is presented in fig. 3 Bottom).
Local Editing. common scenario, the user would like to modify specific object or region, while preserving the rest of the details (i.c., background). For this purpose, utilize the cross-
5

---

Published as a conference paper at ICLR 2023
"cat riding hicvele."
bicycle
Source Image
wio injection
Full Injection
Figure 5: Attention injection through a varied number of difftusion steps. We edit the image by replacing word and injecting the cross rross-attention maps of the source image ranging from 0% (left) to 100% (right) of the steps. Without injection, none of the source content is preserved. while injecting throughout all the steps may over-constrain the geometry. The latter results in low fidelity to the text, e.g,, the car becomes bicycle. The full figure is in the appendix (fig. 17.
attention map layers corresponding to the edited object. In practice, we approximate a mask of the edited part and constrain the modification to be applied only in this local region (lines 11-14 in Algorithm To calculate the mask at step 1, we compute the average attention map M,,, (averaged over steps ,...., of the original word w and the map M of the new word w*. We then apply threshold to produce binary maps, where B(x) == x >> k and A = 0.3 throughout all our experiments. To support geometry modifications of the object, the edited region should include the silhouettes of both the original and the newly edited object, therefore, our final mask is union of the binary maps. Lastly, we use the mask to constrain the editing region (line 13), where @ denotes element- wise multiplication.
Word Swap. In this case, the user swaps tokens of the original prompt with others, e.g., P ="a big bicycle" to P ="a big car". The main challenge to preserve the original composition while also addressing the content of the new prompt. To this end, we inject the attention maps of the source image into the generation with the modified prompt. However, the proposed attention injection may over-constrain the geometry, especially when a large structural modification, such as "car" to "bicycle", is involved. We address this by suggesting a softer attention constrain:
M,
Ed M2.
MM
otherwise,
where is timestamp parameter that determines until which step the injection is applied. Note that the composition is determined in the early steps. Therefore, by limiting the number of injection steps, can guide the composition while allowing the necessary geometry freedom for adapting to the new prompt. An illustration is provided in section 4 Another relaxation is to assign a different number of injection steps for the different tokens in the prompt. If the two words are represented using a different number of tokens, we auplicate/average the maps as necessary using an alignment function as described in the next paragraph.
Prompt Refinement. In another setting, the user adds tokens the prompt, e.g., P ='a castle" to " ='children drawing of a castle". To preserve the common details, we apply the attention injection only over the common tokens from both prompts. Formally, we use an alignment function A that receives a token index from target prompt P and outputs the corresponding token index P None if there isn't 't match. Then, the editing function is:
1 (M))
(Mi))ss
if A(j) = None otherwise.
(Edit (M,, M2MMMM.)..,. t), ==
Recall that the index corresponds to a pixel value, where corresponds to text token. Again, we may control the number of injection steps. This enables diverse capabilities such as stylization, specification of object attributes, or global manipulations as demonstrated in section 4
Attention Re-weighting. Lastly, the user may wish to strengthen or weakens the extent to which each token affects the resulting image. For example, consider the prompt P "a fluffy ball", and assume we want to make the ball more or less fluffy. To achieve such a manipulation, we scale the attention map of the assigned token j* with parameter E [-2, resulting in stronger/weaker effect. The rest of the attention maps remain unchanged. The editing function is therefore:
cc (Me)ss
i-
(Edit (M, M7 t)) (M,)).
otherwise.
As described in section the parameter allows fine and intuitive control over the induced effect. In addition, since fine textures are generated during the super-resolution phase, we observe that this
application can benefit from applying our method also to the super-resolution diffusion model in the case of amplifying attenuating such fine textures, such as "fluffiness" as shown in fig. 7

---

Published as a conference paper at ICLR 2023
Local description
Global description
Souree black
Appurt
old
blossom
Mashatann"
car on side of street."
Figure 6: Editing by prompt refinement. By extending the description of the initial prompt, we perform local or global editing. Additional results are in the appendix (fig. 73 23
"The picnic ready under blossom(() tree."
Figure 7: Text-based editing with fader control. By reducing increasing the cross- Pross-attenioon of specific words (marked with an arrow), we control the extent to which it influences the generation. Additional results are in the appendix (fig. 24
3.3 SELF-ATTENTION
Most models also consist of self-attention layers, which affect the spatial layout and geometry of the generated image as well. However, unlike cross-attention, the interaction that occurs in self- attention layers is only between the pixels to themselves. Therefore, manipulations with respect to specific textual tokens are not feasible. For example, our proposed attention re-weighting and local editing require the matching between the cross-attention maps to the prompt tokens. Another example presented in the appendix (fig. [T]]. where we do not inject the attention of the entire prompt but only the attention of specific word "butterfly". This enables the preservation of the original butterfly while changing the rest of the content. Contrarily, we can't I specify which object should be preserved using only self-attention. Moreover, observe that the self atteetttnn maps provide inferior semantic control compared to the cross-attention. For instance, as demonstrated in the appendix (fig. 17), using cross-attention injection we can swap between apples and oranges by swapping these words the prompt. The same experiment fails when using self-attention which lacks strong interaction between textual tokens and pixels.
Yet, we find that injecting self-attention through small portion (20%) of the steps in addition to cross-attention injection might further help preserve the source content in some cases. And so, we consider it as an additional tool for Prompt-to-Prompt editing. We provide further analysis in appendix
RESULTS In this section, we show several applications of our approach and compare it to other methods.
4.1 APPLICATIONS
Text-Only Localized Editing. We first demonstrate localized editing by modifying the user- provided prompt without requiring any user-provided mask. In fig. [ we generate an image using the prompt "lemon cake". Our method allows us to retain the spatial layout, geometry, and se- manties when replacing the word "lemon" with "apple" (top row). Observe that the background is well- well-preserved including the top-left lemons transforming into apples. On the other hand, naively feeding the model with the prompt "apple cake" results completely different geometry (2nd row), even when using the same randomness in deterministic setting (DDIM). Our method suc- ceeds even for a challenging "pasta cake." " the generated cake consists of pasta layers with tomato sauce on top. In case the user adds new specification, prompt, while allowing the generator to address the newly added words. For example, see fig, 6 where add "old" to the "car", resulting in newly added details over the source car while the background is preserved. Additional results are the appendix (fig. 16 22 and 23
keep the attention maps of the original
As presented in fig. 5 our method is not confined modifying only textures and can modify the structure as well, e.g.. changing a "bicycle" to a "car". We first show the results without cross- attention injection, where changing word leads to entirely different outcome. We then show

---

Published as a conference paper at ICLR 2023
Table 1: User Study results. The participants were asked rate: (1) background / structure preservation with respect to the source image, (2) alignment to the text, and (3) realism.
VQGAN+CLIP
Text2Live
baseline
Ours
(1) Background/ / Structure f
1.84 + 1.11
1.15 + 1.09
3.38 + 1.12
1.64 + 0.64
(2) Text Alignment
2.46 + 1.16
2.89 + 1.22
4.26 + 1.03
4.55 + 0.71
Realism f
1.32 0.70
2.36 + 1.12
4.11 + 0.93
4.42 + 0.82
the resulting image by injecting attention to an increasing number of steps. Note that applying the cross-attention injection in larger number of steps results in greater similarity to the source image. Therefore, the optimal result is not necessarily achieved by applying the injection throughout all steps. This enables us an even better control by changing the number of injection steps.
Global editing. Preserving the composition is not only valuable for local editing, but also an im- portant aspect of global editing. In this setting, the editing should affect all parts of the image, but still retain the original composition, such as the location and identity of the objects. For example, in fig. 6 we preserve the content while changing the lighting. Additional examples are in the appendix (fig. 18 including translating sketch into realistic image and inducing an artistic style.
Fader Control using Attention Re-weighting, While controlling the image by editing the prompt is very effective, we find that it still does not allow full control over the generated image. Consider the prompt "snowy mountain". However, it is quite difficult to describe the desired amount of snow through text. Instead, we suggest a fader control (Lample et al. (017)) where the user controls the magnitude of the effect induced by specific word, as in fig. [ As described in section 3.2 we achieve such control by re-scaling the attention of the specified word. Additional results are in the appendix (fig. 24 and 30).
user may want to control the amount of snow on the mountain.
Different Backhone We use the Imagen (Saharia et al., 2022b) model backbone for most of our experiments and results, exploiting its state-of-the-art synthesis quality. However, our method not limited to a specific model and can be applied to different models as long as they consist of cross-attention layers which are widely used. To validate this, we present results in the appendix 28 29 and 30) using the public and popular Latent Diffusion and Stable Diffusion models (Rombach et al. 2021). As can be seen, our method works well using these models as a backbone, enabling various editing capabilities while preserving the source image content. Further analysis is provided in appendix C.T
(fig. 25 26 27
Real Image Editing. Editing a real image requires finding an initial noise vector that produces the given input image when fed into the diffusion process. This process, known as inversion, has recently drawn considerable attention for GANs (Xia et al., 2021b Bermano et al. 2022), but has not yet been fully addressed for text-guided diffusion models. We show preliminary editing results on real images, based on common inversion techniques for diffusion models. First, a rather naive approach add Gaussian noise to the input image, and then perform predefined number of diffusion steps. Since this results in significant distortions, we adopt an improved inversion approach (Dhariwal & Nichol 202T Song et al.] 2020), which is based on the deterministic DDIM model rather than the DDPM. We perform the diffusion process in the reverse direction, that is xo + IT instead of Tr + Fg. where 2g is set be the given real image. This process often produces satisfying results, as presented in the appendix (fig. 19). However, the inversion is not sufficiently accurate in other tradeoff, where we recognize that reducing the classifier-free guidance (Ho & Salimans| 2021) parameter (i.e., reducing the prompt influence) improves reconstruction but constrains our ability to perform significant manipulations.
cases, as in fig. 20 This partially due to iistortiio-editabblle
To alleviate this limitation, we propose to restore the unedited regions of the original image using a mask, directly extracted from the attention maps. Note that here the mask is generated with no guid- ance from the user, as described in the local editing paragraph (section 3.2). As presented in fig. 21 this approach works well even using the naïve DDPM inversion scheme (adding noise followed by denoising). Note that the cat's identity is well rell-preserved under various editing operations, while the mask is produced only from the prompt itself.
4.2 COMPARISONS
To evaluate our method, we first randomly generate text-based editing examples from predefined text templates, see appendix F for more details. Source text is then fed to the Imagen model to obtain the source image. We compare our results to other text-guided editing methods: (1) VQGAN+CLIP
8

---

Published as a conference paper at ICLR 2023
"Photo of equisel bear enjoys ut the playground."
Source image
VQGAN+CLIP
Tex12Liv
Baseline
Ours
"A bucket full wilb-ppples tomatoes lying on the table."
Source image
Inpainting mask
Blended Ditlusion
Glide
Bascline
Ours
Figure 8: Visual comparison. Top: text-guiee text-guided editing methods (same supervision ours). Bot-
tom: text- rext-guided inpainting methods which rely on an additional input mask (on the left).
(Crosson 2021), (2) Tex22Live (Bar-Tal et al. 2022), (3) Blended Diffussoo Avrahami et al (202bb) and (4) Glide (Nichol et 2021). We also consider (5) a baseline approach where we only replace the source prompt with the target prompt after 20% of diffusion steps using the same random seed.
Qualitative Comparison. As can be seen in fig. g both VQGAN+CLIP and Text2Live may result in severe artifacts when editing highly structured objects, e.g., a squirrel to bear. Our method and Text2Live better preserve the background since both methods estimate mask editing layer. In contrast, the baseline approach produces realistic and meaningful results, but fails to preserve the background. Furthermore, both VQGAN+CLIP and Text2Live require optimization per example which takes 3 and 9 minutes respectively on a GPU. Our method is applied in single diffusion pass which takes up to seconds.
We also consider text-driven inpainting methods which rely given user-defined mask. As can be seen fig. 8 Glide and Blended Diffusion do produce meaningful edits, but fail to preserve the original structure. Note that these approaches limited local changes and cannot handle global edits such as changing the weather in the image. See fig. and 13 in the appendix for more qualitative comparisons.
Quantitative Comparison. In the absence of ground truth for text-based editing, quantitative eval- uation remains open challenge. Therefore, similar to (Bar-Tal al.) 2022), present study in table []] The participants were asked to rate each result in terms of (1) background and structure preservation with respect the source image, (2) alignment the text, and (3) realism. Please see appendix D for more details. As shown, the users preferred our method with regard to all three aspects. Glide and Blended Diffusion were not quantitatively evaluated since they require manual labor to produce the input masks.
We provide additional measures the appendix (table further validate our claims. We evaluate text-image correspondence using their CLIP score, demonstrating competitive results to methods that directly optimize this metric. In addition, we evaluate the perceptual similarity between the original and edited images using LPIPS (Zhang et al.) 2018a) and MS-SSIM (Wang et al. 2003). This shows our capability performing local editing, similar to Text2Live (Bar- Tal et al. 2022 However, CLIP score and perceptual similarity do not reflect our superior quality and realism which are demonstrated the user study.
CONCLUSIONS
In this work, we uncovered the powerful capabilities of the cross-attention layers within text-to- image diffusion models. We showed that these high-dimensional layers have an interpretable rep- resentation of spatial maps that play key role in tying the words in the text prompt to the spatial layout of the synthesized image. With this observation, we showed how various manipulations of the prompt can directly control attributes in the synthesized image, paving the way various ap- plications including local and global editing. This work is a first step towards providing users with simple and intuitive means to edit images and navigate through semantic, textual, space, which exhibits incremental changes after each step, rather than producing an image from seratch after each text manipulation.

---

Published as a conference paper at ICLR 2023
6 ETHIC STATEMENT
Our work suggests a new editing technique for images that are generated using state-of-the-art text- to-image diffusion models. As explained in section 4.1 and appendix D our approach edit real images, although this still remains a more challenging setting. Such manipulation might be exploited by malicious parties to produce fake content in order to spread disinformation. This is known problem, common to all image editing techniques. However, research in identifying and preventing malicious editing is already making significant progress. We believe our work would contribute to this line of work, since we provide a comprehensive analysis of the editing procedure using text-to-image diffusion models.
real photos
ACKNOWLEDGMENTS
We thank Noa Glaser, Adi Zicher, Yaron Brodsky, Shlomi Fruchter and David Salesin for their valuable inputs that helped improve this work, and to Mohammad Norouzi, Chitwan Saharia and William Chan for providing us with their support and the pretrained models of Imagen (Saharia et al. 2022b). Special thanks to Yossi Matias for early inspiring discussion on the problem and for motivating and encouraging to develop technologies along the avenue of intuitive interaction.
REFERENCES
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint arXiv:2206.02779, 2022a.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18208-18218, 2022b.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. arXiv preprint arXiv:2204.02499, 2022.
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word, 2021.
Amit H Bermano, Rinon Gal, Yuval Alaluf, Ron Mokady, Yotam Nitzan, Omer Tov, Oren Patashnik, and Daniel Cohen-Or. State-of-the-ar the architecture, methods and applications of stylegan. In Computer Graphics Forum, volume 41, pp. 591-611. Wiley Online Library, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Katherine Crowson. Vggan + clip, 2021.
https: research. com/ drive/1L8oL www.W//RRCCBPOwOO....... -aa.:kkKKKKKK-PPPPPP--
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.08588, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780 8794, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 19822-19835, 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. 202/ IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 12868-12878. 2021a.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 22873-12883 2021b.
10

---

Published as a conference paper at ICLR 2023
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip- guided domain adaptation of image generators. arXiv preprint arXiv:2108.00990 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to- image synthesis. IEEE transactions on pattern analysis and machine intelligence, 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23: 2721,2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2426-2435, 2022.
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. arXiv preprint arXiv:21 2021.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Mare' Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. Advances in neural information processing systems, 30, 2017.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image genera- tion. Advances in Neural Information Processing Systems, 32, 2019.
Ron Mokady, Omer Tov. Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel. Daniel Cohen-Or, and Michal Irani. Self-distilled stylegan: Towards generation from internet photos. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pp- 1-9, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint ar 10741, 2021.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. arXiv preprint arXiv:2103.172493 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.000 103.00020. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(14)): 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp.
PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2 2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11887, 2022b.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Ming Tao, Hao Tang, Songsong Wu, Nieu Sebe, Yuan Jing, Fei Wu, and Bingkun Bao. Df- gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.0m 05865, 2020.
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10748-10757, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Zhou Wang, Ecro P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thräty-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1402. Ieee, 2003.
Weihao Xia, Yujiu Yang. Hing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2256-2265, 202
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey, 2021b.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprini arXiv:21 10.04627, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-ime image generation. arXiv preprint 1078,9,9 2022.
12

---

Published as a conference paper at ICLR 2023
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 386-595,44 2018a.
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with hierarchically-nestee adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1199-6208, 2018b.
A BACKGROUND
A.1 DIFFUSION MODELS
Diffusion Denoising Probabilistic Models (DDPM) Sohl-Dickstein et al. (2015); Ho et al. (2020)
generative latent variable models that aim to model distribution pe( that approximates the data distribution ((0)) and casy to sample from. DDPMs model a "forward process" in the space This process is Markov chain starting from Ho where gradually add
of zo from data noise []] noise to the data to generate the latent variables 21,...,, .T E X. The sequence of latent variables therefore follows g( Tt xo) = II g(xt It-1),, where step in the forward process defined as a Gaussian transition g(x: xt-1) == N(xx; vT B, 3,7) parameterized by a schedule Bo.-.., 3r E 1). When T is large enough, the last noise vector 2T nearly follows isotropic Gaussian distribution.
An interesting property of the forward process is that one can express the latent variable xa directly as the following linear combination of noise and to without sampling intermediate latent vectors:
It varzo + vI a.w, w ~ NO,,
(2)
where = (1 (1-
In order to sample from the distribution g(xo), we define the dual "reverse process" p(x+-1 1 ax) from isotropic Gaussian noise 27 to data by sampling the posteriors ( z,,) Since the intractable reverse process g( xx) depends on the unknown data distribution q(xO), we approximate it with parameterized Gaussian transition network po(# xx) == N(1-11
ful [2,, t), Ee (Zt, t)). The (ug t) can be replaced (Ho et al. 2020) by predicting the noise c2(22,
added to zo using equation
Under this definition, we use Bayes' theorem to approximate
1 1 5t vI VI-ai
on (10t, _e)
3e
po(a,, 1)
(3)
Vou
at
Once we have a trained ca 1), we can using the following sample method
ft (x+,,)) + 022, NO.
(4)
We can control 0, of each sample stage, and in DDIMs (Song al. 2020) the sampling process can be made deterministic using 0 in all the steps. The reverse process can finally be trained by solving the following optimization problem:
min L(0) = min
[w c0 1)
teaching the parameters to fit g(xo) by maximizing a variational lower bound.
A.2 ATTENTION LAYERS IN TEXT TO IMAGE DIFFUSION MODELS
Imagen, Latent Diffusion, and Sta- ble Diffusion. We describe here only high-level description of each model and its attention layers that are relevant to our method. Note that these models condition on the text prompt in the noise prediction of each diffusion step through two types of attention layers: i) cross-attention layers. ii)
We implement our method on three different diffusion models:
This process called "forward" due its procedure progressing from Jo JT.
13

---

Published as a conference paper at ICLR 2023
hybrid attention that acts both as self-attention and cross-attention by concatenating the text embed- ding sequence to the key-value pairs of each self-attention layer. Our method only intervenes in the cross-attention part of the hybrid attention. That is, only the last channels, which refer to text tokens, are modified in the hybrid attention modules.
Imagen. (Saharia al. 2022b) consists of three text-conditioned diffusion models and language model: A text-to-image 64 x 64 model, two super-resolution models 64 x 64 - 256 x 256 and 256 256 + 1024 1024 and pre-trained T5 XL language model Raffel
al. (2020). These predict the noise (=t,,, 1) via U-shaped network, for / ranging from 7 to 1. Where 2, is the latent vector and is the text embedding of the language model. We highlight the differences between the three diffusion models:
64 64 starts from random noise, and uses the U-Net as in (Dhariwal & Nichol 2021). This model is conditioned on text embeddings via both cross- cross-attention layers at resolutions [16, 8] and hybrid-attention layers at resolutions [32, 16, 8] of the downsampling and upsampling within the U-Net.
64 x 64 - 256 x 256- conditions on a naively upsampled 64 x 64 image. An efficient ver- sion of a U-Net is used, which includes Hybrid attention layers in the bottleneck (resolution of 32).
256 x 256 + 1024 1024 - conditions on a naively upsampled 256 x 256 image. An efficient version of a U-Net is used, which only includes cross-attention layers in the bot- tleneck (resolution of 64).
Latent Diffusion. Latent Diffusion Model (LDM) (Rombach et al. 2021) is substantially different from Imagen. First, to reduce memory consummation, LDM operates in the latent space a pre- trained VQGAN et al. (22TT; Esser et al. (202 la). This reduces the spatial size of an input image from 256 x 256 to a quantized latent space of size 32 x 32 with 4 channels. Second, the language model is trained from scratch with the main diffusion model and consists of 32 transformer layers. For the diffusion process, a U-Net is used as in (Dhrriall & Nichol, 2021), which consists of self- attention layers followed by text-conditioned cross-attention layers resolutions 32, 16, 8 and 4.
Stable Diffusion. Stable Diffusion (SD) is an improved version of LDM which is trained on higher resolution with more resources and data. The latent space is of size 64 x 64 with 4 channels, which after decoding results in an image of size 512 x 512. SD uses pre-trained CLIP model (Radford) al. 202T) for the conditioned text embedding, and consists of self-attention layers followed by lext-conditioned cross-attention layers at resolutions 64, 32, 16 and 8.
B SELF-ATTENTION IN SEXT-CONITTIOED DIFFUSION MODELS
An interesting question is the role of self-attention maps. In particular, compared to cross-attention, how well it reveals the structure of the generated image, and how its injection affects the image generation under our settings.
Similar to the cross-attention maps, we found that self-attention maps are correlated the structure and different semantic regions in the image. As can be seen in fig. | the self-attention maps of different pixels highlight the close region of the pixel in addition regions in the image that contain the same semantic content. For example, a pixel on the crust of the pizza attends to other pixels on the crust. In addition, if look at the top principle components of the self-attention maps, we can clearly identify the layout of the generated image, as previously shown in (Tumanyan et al., 2022) for different model. However, since the self- self-attention maps are not correlated to specific words, these provide inferior control compared to cross-attention maps. For instance, it is much more challenging to find map that highlights only the pepperoni using self-attention.
Next, we inject the self-attention maps of a source image during the generation of an image con- ditioned on another target prompt. Notice that the source and target prompts might be unaligned in this scenario. Such examples are shown in fig. 12] where we apply self-attention injection for a gradually increased number of diffusion steps. As we can see, the attention maps drastically af- feet the resulting images such that injecting the maps for more than 50% steps suppresses almost any connection to the target prompt. Interestingly, the self-attention maps can also determine the color palette in the image. Since the self-attention injection may restrict the editing capability of our
14

---

Published as a conference paper at ICLR 2023
Average cross-attention maps
Synthesized
"image
of
an
clophant
playingg
tenais"
Average eelf-attention maps of individual pixels
Top principle components the self- attentioo maps
Average attention maps
Synthesized image
"pepperoni
pizza
next
orange
juice
Average aelf-attention maps of individual pixels
Top principle companents of the self-attention maps
Figure 9: Visualization of cross and self-attention. The top row for each example illustrates the average cross-attention maps for the given prompt. The second shows the self-attention maps with respect to different pixels (marked in green). The third row shows the principle components of the self-attention maps. All examples present the attention maps resolution 16 x 16 after averaging across diffusion steps, different layers, and attention heads.
method, we use self-attention injection for up to 20% of the diffusion steps. We found that this may improve the source background preservation some cases.
C ADDITIONAL RESULTS
Additional quantitative results are provided in table 2
Full figures for fig. 5 and are in fig. and 13 respectively. Additional qualitative comparisons are provided fig. and
15

---

Published as a conference paper at ICLR 2023
In fig. 16 we do not inject the attention of the entire prompt but only the attention of specific word "butterfly". This enables the preservation of the original butterfly while changing the rest swap between
of the content. As demonstrated in fig. 17 using cross-attention injection apples and oranges by swapping these words in the prompt. The same experiment fails when using self-attention which lacks a strong interaction between textual tokens and pixels.
Additional global editing results are presented in fig. [8 illustrating a translation of sketch into a photo-realistic image and inducing an artistic style. Examples for editing of real images provided in fig. 19 20
and 21
We provide additional visual examples for different editing operations using method: fig. show word swap results, fig. show adding specification to an image, and fig. 24 show attention re-weighting.
C.I DIFFERENT BACKBONES
Results for the Latent Diffusion and Stable Diffusion models in fig. 25 26 We observe that text-based replacement and refinement operations work well for all three models. However, notice small difference between the three models in the Fader Control using Atten- tion Re-weighting. Visual examples of this application are presented in fig. 24] 27 and [30 using Imagen, Latent Diffusion and Stabe Diffusion respectively. As can be seen, when using Imagen as the backbone, our method produces high-quality results and can even handle delicate changes such as reducing the "cubic" appearance of sushi. On the other hand, applying our method with Stable Diffusion may result in unexpected artifacts. For example, when reducing the attention to the word "night" (last example in fig. 30). not only the time of the day changed but also the dark skies turns into trees.
28 and 30
We hypothesize that this difference is the result of using different language models for text embed- ding. Imagen uses a T5 language model that is trained using an unsupervised language objective of span masking. Stable Diffusion uses CLIP which trained with a multi-modal constructive objective. Lastly, the Latent Diffusion language model is trained with the same reconstitution objec- tive as the diffusion model. Therefore, we suggest that the text embedding of T5 better represents disentangled information and yields superior results.
D LIMITATIONS
While we have demonstrated semantic control by changing only textual prompts, our technique is subject to a few limitations. First, the current inversion process results in a visible distortion over some of the test images, see fig. 20) Moreover, the inversion requires the user to come up with a suitable prompt which could be challenging for complicated compositions. Note that the challenge of inversion for text-guided diffusion models is an orthogonal endeavor to our work, which would be studied in the future. Second, current attention maps are of low resolution, as the cross-attention is placed in the network's bottleneck. This bounds our ability to perform more precise editing. To alleviate this, we suggest incorporating cross-attention also in higher-resolution layers. We leave this for future work as it requires analyzing the training which is out of our scope. Third, our method requires setting the timestamp parameter for attention injection and the scale parameter for attention re- re-weighting. Tuning these usually requires roughly a minute for the Imagen model. However, our method is not highly sensitive these, and using constant timestamp produces satisfying results in most cases. cases.Furttin Furthermore, we believe that future works will reduce the inference time of these models, so tuning the parameter will be quicker and more intuitive. Finally, we recognize that our method cannot be used for large structural changes in the image, like changing the pose of an animal, move objects or changing the number of objects in the image, see examples in fig. 10 We leave this kind of control for future work.
E USER STUDY
32 participants answered our user study. Each was asked to evaluate 18 randomly selected Prompt- to-Prompt examples for each method. The examples were given in random order and were divided into three parts: (A) consists of 6 replacement examples using templates 1 and 2 (see appendix F). (B) consists of 6 local refinement examples using templates 3 and 4. (C) consist of global refine-
16

---

Published as a conference paper at ICLR 2023
"A dog playing the field."
"A living room with two chairs."
"A child holding bunny doll."
Source image
playing sleeping
Souree image
four chairs..."
Source image
..mmiiing bunny doll."
Editing Failure Cases. Since Prompt-to toomppp preserves the overall structure of the source image, it fails when large structural modification required. For example, changing the pose of the dog from "playing" to "sleeping" (on the left) changing the number of chairs (middle). In addition, our method is limited by the semantic understanding of the diffusion model, for example, on the right, add the refinement "smiling bunny doll" that made both the bunny and the child smile.
Figure 10:
ment examples using templates 5 and 6. For each example the user was asked to rate the image on
1 5 scale (higher is better) with respect to the following questions:
(1) How well does the right image preserve the structure and the background of the left image? Consider the preservation of properties that are not specified by the text above the images.
(2) How well does the right image match the text description above it? Specifically, consider the highlighted text.
(3) Rate the overall realism and quality of the right image.
See fig.
3 for screenshots.
F EVALUATION PROMPTS
We use the following prompt templates to generate the evaluation data:
Template.1: "Image of <A_RPCS inside <B.CONST>."
Select_A ["apples", " "oranges", "chocolates", "kittens", "puppies", "candies") Select. ["box", "bowl", "bucket", "nest", "pot"]
Template 2: "A <A RPC> full of < CONST> is lying the table." Solect.A ["box", ", "bowl", "bucket", "nest", "pot"] Select.B ("apples". "oranges". "chocolates". "kittens", "puppies", "candies")
Template_3:
<A_RPC, <CONST>."
select.from.a ["cat", "dog". "lion" "camel", "horse", "bear", "squirrel", "clephant". "zebra". "giraffe". "cow"l select.from ["seating in the field", "walking the field", "walking in the city", "wandcring around the city". wandering in the streets". "walking in the desert", "seating
desert". "walking "walking in the desert". "seating in the desert", "plays at the playground", "enjoys
forest", "seating
forest".
playground"]
Template 3:
"Photo a <A CONST> <B ADD> with CONOSTS> on it."
Select.A ["treee, "shrub", "flower", "chair", "fruit"] select.B ["made of candies", "made of bricks". "made of paper". "made of clay". "made feathers
"made
Select C ("bug", "butterfly" ", "bee", "grasshopper", "bird"]
Template 4: "Image of <A <B CONST the side of the road." Select A ("wooden", "old", "crashed", "golden", "silver". "sport", "loy"I select_B ("car", "bus", "bicycle". "motorcycle". "scooter". "van"
Template 5: "A landscape Image of <A CONST> ADD?" Select.A
["a river", "a lake" " "a valley" "mountains". the valley".
"a forest". "a river
"a vlilage on mountain".
"a waterfall between the mountains". "the cliffs the desert"]
select_B ["in the winter" ", "in the autumn" "at night" "at sunset". "at sunrise". fall" rainy day "in clondy day". evening")
17

---

Published as a conference paper at ICLR 2023
Source image
Source Prompt: "Photo of cat riding bicycle."
bicycle motoreycle
bicycle train
chicken
fish
W.O. cross cross-attention injection
Full cross-attention injection
Figure 11: Attention injection through a varied number of diffusion steps. Top: source image and prompt. each row, we modify the content of the image by replacing a single word in the text and injecting the cross-attention maps of the source image ranging from 0% (left) 100% (right) of the steps. Without our method, none of the source image content is guaranteed to be preserved. On the other hand, injecting the cross-attention throughout all the steps may over-constrain the geometry, resulting in low fidelity to the text.
Template_6: "CASCONST>> in the <BLADDE
Select.A = ("Heavy traffic". "The houses". "The buildings". "Cyeling".
the station select_B ("snowy" "flooded" "blossom" ", "moder". "historic", "commercial" "colorful"]
"The tram passing"
We generate 20 random examples using each template where the tokens <CONST>, <RPC> and <ADD>> where modomly replaced with one item conrepooddng selection list below each template.
<ONST>> stands for phrase that used both source and larget prompt.
RPC> stands for phrase that different between the source and target.
ADD stands for refinement phrase which is only replaced in the target and omitted the source prompt.
18

---

Published as a conference paper at ICLR 2023
Souree image
Source prompt: "A sailing boat near a castle."
Target prompt: "An elephant in the field."
La
Target prompt: "A bodroom."
Target prompt: "Vanilla ice cream with marshmallows."
Target prompt: "I city skyline.."
W.O. self-attention injection
Full self-attention injection
Figure 12: Self-attention injection through a varied number of diffusion steps. each row, we conditioned the image generation target prompt and inject the self-attention maps of the source image ranging from 0% (left) to 100% (right) of the diffusion steps.
Table 2: Additional quantitative results. We measure text-image correspondence using CLIP (Rad-) ford et al. 2021), demonstrating competitive results to methods that directly optimize the CLIP score. In addition, we evaluate the similarity between the original and the edited images using the L.PIS (Zhang et al. (2018a) perceptual distance and MS-SSIM (Wang al. (003). This show our capability of performing local editing, similar to Tex12Live (Bar- Tal et al. 2022).
CL.IP score
MS-SSIM
I.PISS +
VQGAN+CLIP
0.282 + 0.04
0.27 + 0.046
0.64 0.05
Text2Live
0.247 + 0.04
0.82 0.065
0.25 0.05
baseline
0.253 + 0.03
0.69 0.13
0.35
Ours
0.253 + 0.04
0.81+0.11 0.11
0.22 + 0.1
19

---

Published as a conference paper at ICLR 2023
"A car the side of the street."
gelden
Amerriuu
Source image
....ybberpunkk car.."
Immomine car.."
Jutuisti cur...
". emergency can.."
Local description Global description
snoryy
evening
Figure 13: Editing by prompt refinement. By extending the description of the initial prompt, we can make local edits to the car (top rows) or global modifications (bottom rows).
20

---

Published as a conference paper at ICLR 2023
"Photo of a eut camel seating in the forest."
"Image bowl with eranges chocolates." "
"Photo of a flower made of candies with a butterfly on it."
"Image of golden scooter the side of the road." "
landscape image of lake at sunset."
Source image
VQGAN+CLIP
Text2Live
Baseline
Ours
Figure 14: Additional comparisons to text-guided image editing. Similar to ours, these methods do require a user-provided mask.
21

---

Published as a conference paper at ICLR 2023
"Photo of lion zebra seating in the forest."
"Photo eut bear walking in the desert."
"A plate full of cookies puppies is lying on the table."
"Photo of flower made of feathers with bug it."
"Image of Lego motorcycle on the side of the road."
Source image
Inpainting mask
Blended Diffusion
Glide
Ours
Figure 15: Additional comparisons to text-guided in-painting methods. Unlike our method, these techniques require auxiliary segmentation mask which is provided by the user.
22

---

Published as a conference paper at ICLR 2023
Cross-attention injection
oo viatin.
priste
Choclate"
AB muttes.
.o gizzs"
Self- Self-attention injection
AB dite."
"..00 vicin."
.0 & Choclate."
1.00 pizza
"LOB DIRRE
"A photo of butterfly on..."
Figure 16: Object preservation and replacement using cross and self attention injection. Top: by injecting only the cross-attention weights of the word "butterfly" taken from the top-left image preserve the structure and appearance of single item while replacing its context (i.e., background). Bottom: using only self-attention injection we can't specify which object should be preserved, therefore, modifying the background while keeping the butterfly is more challenging.
23

---

Published as a conference paper at ICLR 2023
"apples"
"oranges"
Source images
Sell-attention injection
Cross-attention injection
"oranges" "
"kittens"
Source images
Self stteettion injection
Cross attention injection
Object replacement using cross-attention injection and self-attention injection. Cross Cross-attents aatt-ttttnnonn injection better preserves the semantic relation between the generated image and the text prompt. Top: using cross-attention injection (third row) we can swap between apples and oranges in the source image by swapping these words in the prompt "apples and oranges are the table.". The same experiment fails when using self-attention which lacks a strong interaction between textual tokens and pixels. Bottom: cross-att coss-attention injection (6th row) better preserves the distinct elements in the image when replacing the word "oranges" with "kittens" the sentence "a basket with oranges the counter."
Figure 17:
24

---

Published as a conference paper at ICLR 2023
"drawing of..."
"photo of..."
source image "photo of..."
"relaxing photo of..."
"dramatic photo of..."
"n..in the jungle."
in the desert."
marrs.
"painting of..."
source image
wateeroooo....
charoocal...
"impressionism....
"futuristic...
"neo classical..."
"A waterfall between the mountains."
Figure 18: Image stylization. By adding style description to the prompt while injecting the source attention maps, create various images in the new desired styles that preserve the structure of the original image.
"A black walking grass."
image
roconstructod
"..sext Rowers."
comes down."
motthe is walthing."
painting of.."
"Landscape image valley..."
real image
reconstrncted
".aa Gill.
".. winter
"_. suntise."
"..i6 night."
Figure 19: Editing of real images. On the left, inversion results using DDIM Song et al. (2020) sampling. We reverse the diffusion process initialized on a given real image and text prompt. This results in latent noise that produces approximation to the input image when fed to the diffusion process. Afterward, on the right, we apply our Prompt-to- tooprop technique to edit the images.
Real imuge
Recunstrusted
Real image
Reuonstracted
Real image
Reconstrasted
Figure 20: Inversion Failure Cases. Current DDIM-baser -bsed inversion of real images might result in unsatisfied reconstructions.
25

---

Published as a conference paper at ICLR 2023
"image of cat wearing floral shir.""
"shirt"
Input image prompt
noised
denolsed attention
blended
SR
different seeds
Figure 21: Mask-based editing. Using the attention maps, preserve the unedited parts of the image when the inversion distortion is significant. This does not require any user-provided masks, as we extract the spatial information from the model using our method. Note how the cat't 'ss identiry is retained after the editing process.
26

---

Published as a conference paper at ICLR 2023
"Photo of riding bicycle."
source image
cat
dog
cat
chicken
squirel
elephant
"Photo house with a flag on a mountain." TM
source image
house
>>
hotel
house
tent
house > car
house > free
"A basket full of apples."
source image
basket
basket
basket pot
hasket nest
apples
oranges
apples
>
chocolates
apples
>
cookies
apples
kittens
apples > smoke
A ball between two chairs the beach.
SOUreC image
ball
bucket
ball
palace
ball
busket
ball
turtle
Figure 22: Additional results for Prompt-to-Promp editing by word swapping using the Imagen model (Saharia et al. 20226)..
27

---

Published as a conference paper at ICLR 2023
photo wearing sunglasses having drink." "
g.egekky sunglass
"...ber drink."
Source insage
squared sunglesses..."
"..colorful sunglasses...
" wofffe drink."
"A photo of butterfly on flower."
wither Hower."
or.oggaaai flower"
image
spiley
candies."
"A mushroom the forest."
dry mushroom....
Source image
"...in the wet forest."
"Line nrt mush. "A neon mushroom....
Figure 23: Additional results for Coompttto-Promptt editing by adding specification using the Ima- gen model (Saharia et al. 20226)..
28

---

Published as a conference paper at ICLR 2023
leopard slceping(1) cake next to an apple."
"A smiling( teddy bear." "
"Photo of a cubicc v sushi."
photo birthday( cake next apple."
"My colorful( bedroom." "
"Photo of field of poppies night(4)."
Figure 24: Additional results for Prompt-to-Promp editing by attention re-weighting using the Im- agen model (Saharia et al. 2022b).
29

---

Published as a conference paper at ICLR 2023
NOX
YO.
NDX
GO
"A painting of squirrel cating a a-buages pizza.
bench pile books magazines top.
"Banknote portrait a 4em horse."
"Seeit Turtie the middle forest. Aftemoon light."
"A bowl with applas. snacks lable."
"Photo bustecily hee flower."
"A phoro of a dog wearing a 4emm dotted shirt."
"A photo z-ows dog wearing a blue ric."
"A photo cat playing ahew domino
coltoe-femme
bouquet enlips daisies on table."
"A inflated on ground."
agedt
painting Hee hear cating apple."
"A driving beach
"Soup with sice noodles."
"Watercolour painting photo of a latent
"A photo an astronaut riding a hosce- camel.."
"A castle made out of send corn."
"A knowsaa scarecrow in the garden."
"Lamon Apple cake on the table."
"Image of candies toys inside box."
Figure 25: Additional results for Prompt-to-Prompl editing by word swap using the Latent Diffusion Model (Rombach et al., 202T).
30

---

Published as a conference paper at ICLR 2023
beautiful bouquet talips colour red and yellow."
"A school driving street."
"A car driving flooded street.
"A lanckcape with lake between mountains at sunset."
"Pizza with mushrooms."
"A wooden bike the yard."
"A fashion sketch of an evening dress with long sleeves."
"A speeding race car is driving on the heach."
"A big yellow apple on table"
"My bicycle are Vegas,"
banknote portrait
"A painting vase
small clay bunny with big smile."
"My bicyele the blossom.
"Photo of landscape with river and moun- tains sunrise."
"A TV screen with many burnt pixels."
"A stove outside creating smoke cloud above."
"A bear with yellow sunglasses and drink."
"A photo of dog wearing floral shin.""
"A landscape photo of hurbor in the storm.. ?
"The scooter at the city winter."
Figure 26: Additional results for Coompt-to-Prompee editing by adding specification using the La- tent Diffusion Model (Rombach et al. 2021).
31

---

Published as a conference paper at ICLR 2023
"A photo a blossom (4) tree."
"A landscape with a snowy (1) mountain." "
"A photo of the ancient (4) city."
crahsed (v) car."
puffy
piff
piff
DYHo
DEdo
"My puffy (4) shirt."
photo a poppy field at night (4)."
Figure 27: Additional results for Coompt-to-Prompl editing by attention re-weighting using the La- tent Diffusion Model (Rombach et al. 2021).
32

---

Published as a conference paper at ICLR 2023
IO 2000
AA painting of squistel cat eating a burger."
"A bench with many books magazines on
"Banknote portrit of moouee horse."
car driving on the beach road.""
the living room."
"A defleted Inflated tire on the ground."
"A fachion B. sketch of an evening dress of ovening dress."
"A huge lnansincent musirromm avocado in the middle forest. Aftemeon light."
"A kanguuoo deer in pub eating sushi. DSLR
A papporans mushroom pizza on table."
"A stove over pile of diverse Landom-heese sports objects. Low lighting image."
"An evil robot bolding a-werd broom."
origami bostle cup."
"A vase sotton-tennis
"A photo cat playing domino
"Photo of clgg cal in the street."
"A piano made out of Logo cubes." "
"A painting of squirrel eating burger
"A beautiful bouquet 4alips daisies."
"An orange lable."
"Image of comndee mints insäde
Figure 28: Additional results for Comppt-to-Prompee editing by word swap using the Stable Diffusion Model.
33

---

Published as a conference paper at ICLR 2023
"A beautiful bouquet talips of the colour red and yellow.
"A big yellow apple on table"
"A wooden bike in the yard."
"A bridge made of rope between two cliffs."
"A dangerous bridge missing steps between tWo cliffs."
"A speeding race driving on the beach."
"A fashion sketch of evening dress with long sleeves,"
"A painting of squirrel jumping over metal fence with spikes."
"A huge transucent mushroom in the middle forest. Afternoo light." !
"A painting of lilies in the style of Van Gogh."
"A pepperoni pizza with mushroom and olive toppings table"
"A banknote portrait of a mouse."
clay bunny big smile."
recliner sofa living room
"A deflated and ripped up the ground."
"A vase filled with cotton and metal balls."
"A stove outside creating smoke cloud above."
"A TV screen with many burnt pixels."
B
78
"An of with alphubet soup crack English
"Byeglasses the desk reflecting strong glare."
"Image of cundies covered with chocolate inside box."
Figure 29: Additional results for Prompt-to trompt-o--Prompt editing by adding a specification using the Stable Diffusion Model.
34

---

Published as a conference paper at ICLR 2023
"A photo of a blossom (V) tree."
"A landscape with a snowy (4) mountain."
"A photo of the ancient (4) city."
"A crahsed (V) car."
"A smiling(1) ) leddy bear."
"A photo of a poppy field at night (4)."
Figure 30: Additional results for Cooppttoo-roompee editing by attention re-weighting using the Sta-
ble Diffusion Model.
35

---

Published as a conference paper at ICLR 2023
Local Editing
Global Editing
Photo horze seating in the forest.
A landscape image of a village on mountain at sunset
How well does the right image the structure and the background left image? Consider preservation of
well the right image preserve and the content of left Image? Conelder preservation
properties that specified by the text above the images
properties specified by the the images.
(Poor preservttio)
O'zor presenatio))
O
[Dacalent proarmetion)
6/Exeellent
How well does the right image match the text description above it?
How well does the right image match the text description above it?
Specifically consider the highlighted text.
Specifically consider the highlighted text.
(Towech moech at)
Squaedlent mindree)
Rate the overall realism and quality right image."
Rate the overall realism and quality of the right image.
Very slty
5iery High qualtt
Figure 31: Screenshots from our User study. The participants were asked to evaluate: (1) back- ground, structure, and content preservation with respect to the source image, (2) alignment to the text, and (3) realism. The study evaluates both local and global editing.
36


Reviews:
[{"review_id": "_804CBiejQa", "reviewer": "Reviewer_kTNM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. While large-scale language-image (LLI) models are collecting attention to generate an image from text, they have limitation to control by only text. The paper  propose to control the attention maps of the edited image by injecting the attention maps of the original image along the diffusion process. The paper shows diverse results by by editing the textual prompt only.", "summary_of_the_review": "The paper tries to solve an issue of controlling the image by text alone in image editing based on large-scale language-image models. By analyzing the cross-attention layers, the paper nicely shows a way to control the image by text alone. Diverse results show the effectiveness.", "strength_and_weaknesses": "Strength\n1. The paper enables the user to control the generated image by text alone, by keeping the structure.\n2. The proposed framework allows the user to control the extent.\n\nWeakness\n1. Failure cases are not shown for future work. There would be some failure cases or unsolved types of text edition. If the paper can discuss this aspect, the paper would be more appealing to readers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yq_4OceS4v", "reviewer": "Reviewer_vsUX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to only use text to achieve local image editing based on recent text-to-image models. More specifically, they propose to use cross-attention to connect each word and image regions and get rid of user-provided masks. Many interesting applications have been proposed and the results are very promising.", "summary_of_the_review": "This paper proposes a new method to only use text to achieve local image editing based on recent text-to-image models. \nThe method is novel and technically sound. And the author provided various new applications which further prove its effectiveness.\n\nI think this method can also inspire many interesting future work in this area.", "strength_and_weaknesses": "Strengths\n1. The proposed task of only using text for local image editing is very interesting and of good practical values.\n2. The authors proposed to use cross attention to connect word and image regions, which is novel and reasonable. Although attention maps have been widely used in other tasks, I think it is quite appropriate here.\n3. Many interesting applications have been provided. All results are promising and clearly show this method's advantages.\n4. The paper writing is clear and easy to follow.\n\nWeaknesses\n1. Maybe some ablation study would be better. \nFor example, does the proposed method affected by different noise sampler? \n\n2. For those tasks that only local edits are needed, is it possible to only process local area and accelerate the process?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HPYCi4f72LT", "reviewer": "Reviewer_uigg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, an approach to controllable image editing with large-scale text-to-image diffusion models is proposed.\nMore specifically, an approach is presented that first analyzes the cross-attention (and self-attention) layers in the U-net backbone of state-of-the-art text-image diffusion models. It is then shown that these layers capture most of the structural information about the synthesized images.\nThis opens up a number of modification possibilities by directly modifying these attention maps. The paper demonstrates these modification possibilities using, among others, content modification and local semantic editing.\nThis overcomes a problem that arises when using \"vanilla\" prompt-based modifications, where small changes in the prompt can lead to very different synthesis outcomes.\n\n\n", "summary_of_the_review": "This is a very good empirical work with practical implications that greatly extends the applicability of modern text-to-image systems. The only major drawback is the fact that the method was mainly tested and evaluated on a closed-source system, which makes reproducibility difficult. However, since the method has also been validated on open systems, I am happy to recommend its acceptance for ICLR 2023.\n\n", "strength_and_weaknesses": "This is a strong empirical paper with convincing experimental results for editing with text-to-image diffusion models.\nThe finding that most of the object structure is stored in the cross-attention maps is interesting and has practical implications.\nSeveral methods are presented for obtaining finer-grained, local control when editing text.\nFinally, the manuscript is clearly written, easy to understand, and provides very good visual explanations.\n\nA few questions that I had while reading the paper: \nHow much control is lost by only applying the model on the 64x64 base model? Is this different in models using a diffusion model for upscaling or the decoder of an autoencoder? Does the \"attention caching\" lead to an improvement in sampling speed?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "80jkYkymYQ", "reviewer": "Reviewer_5o2W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a text-driven image editing technique to refine images generated from large-scale text-to-image diffusion models. The paper relies on a crucial observation: Just modifying a text prompt when calling the generator results in structurally and compositionally very different images, even if the text prompt is modified only slightly and even if the same random seed is used. However, we can use the attention maps of the cross attention layers (that attend to the text tokens) corresponding to the initial image, and inject them into the generation process when generating a new image with a modified prompt. Now the new image will correspond structurally and compositionally to the original one, while also reflecting the new text prompt. This allows the authors to demonstrate rich text-driven editing capabilities. The method can be combined with masking schemes to localize the edits and re-weighting techniques to give more or less weight to certain words in the prompt. The approach is compared to several baselines and shows favourable results.", "summary_of_the_review": "In summary, I think this is a strong paper, which I recommend for acceptance. The methodology is based on a smart idea and novel, and the results are appealing. The paper is of high-quality throughout and well written and clearly presented. The paper has a couple of small weaknesses and the authors have mostly been transparent about that, but I do not think there are any major flaws. I believe the method could find practical use in the generative modeling community, for instance for producing generative art with even finer control.", "strength_and_weaknesses": "**Strengths:**\n\n- The proposed idea, modifying and re-using the attention maps of a diffusion model from one image, when generating a new one, is simple, yet elegant and novel. The approach makes intuitively sense and works well in practice.\n- Generally, the paper is well-written, well-motivated and easy to follow (also see below). The work is also appropriately put into the broader context in the literature.\n- Experimentally, the editing results are quite strong and the original structure and composition of the edited images is usually indeed well preserved when a new text prompt is used for editing. The fact that this is purely text-driven and yet allows fairly fine control is appealing.\n\n**Weaknesses:**\n\n- The work is only partially reproducible. The Imagen model is not available to the public and cannot be re-trained easily. The results on Stable Diffusion and the Latent Diffusion Model can be worse in some cases (see Appendix D.1).\n- The approach comes with some hyperparameters. For instance, to achieve best results the attention map injection should only happen up to a certain time stamp during the iterative synthesis process. It seems like this time stamp would need to be tuned for each example separately, which can be costly, since running the diffusion model synthesis process repeatedly is expensive. This may not be ideal for interactive, real-time applications.\n- Editing given real images is difficult, because this first requires an inversion process as well as finding an appropriate text prompt that could have generate the given image (this is discussed in the paper).\n- As also pointed out by the authors themselves, other minor weaknesses include that the attention maps that are modified are in the low resolution layers of the U-Net, which prevents very fine-grained editing. Furthermore, editing corresponding to large-scale object movement is not possible with the proposed technique.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%