Diffusion Self-Guidance for Controllable Image Generation
Authors: Dave Epstein
Allan Jabri
Ben Poole
Alexei A Efros
Aleksander Holynski
Venue: NeurIPS 2023 poster (2023)
Keywords: generative models
image editing
diffusion
guidance
TL;DR: Use attention and features from the denoising forward pass to guide diffusion sampling - change object size, position, and appearance without affecting the rest of the scene, and much more!

Abstract: Large-scale generative models are capable of producing high-quality images from detailed prompts. However, many aspects of an image are difficult or impossible to convey through text. We introduce self-guidance, a method that provides precise control over properties of the generated image by guiding the internal representations of diffusion models. We demonstrate that the size, location, and appearance of objects can be extracted from these representations, and show how to use them to steer the sampling process. Self-guidance operates similarly to standard classifier guidance, but uses signals present in the pretrained model itself, requiring no additional models or training. We demonstrate the flexibility and effectiveness of self-guided generation through a wide range of challenging image manipulations, such as modifying the position or size of a single object (keeping the rest of the image unchanged), merging the appearance of objects in one image with the layout of another, composing objects from multiple images into one, and more. We also propose a new method for reconstruction using self-guidance, which allows extending our approach to editing real images.

Diffusion Self-Guidance for Controllable Image Generation
Dave Epstein 1.2
Allan Jabri'
Ben Poole'
Alexei A. Efros' Google Research
Aleksander Holynski 1.2
UC Berkeley
Abstract
Large-scale generative models are capable of producing high-quality images from detailed text descriptions. However, many aspects of an image difficult impossible to convey through text. We introduce seff-guidance, a method that provides greater control over generated images by guiding the internal represen- tations of diffusion models. We demonstrate that properties such as the shape, location, and appearance of objects can be extracted from these representations and used to steer the sampling process. Self-guidance operates similarly to standard classifier guidance, but uses signals present in the pretrained model itself, requiring no additional models or training. We show how simple set of properties can be composed perform challenging image manipulations, such as modifying the position or size of specific objects, merging the appearance of objects in one image with the layout another, composing objects from multiple images into one, and more. We also show that self- gelf-guidncce can be used for editing real images. See our project page for results and an interactive demo:
ml/selfiguidance
Introduction
Generative image models have improved rapidly in recent years with the adoption of large text-image datasets and scalable architectures [7, 10, 12, 27, 28, 32, 34, 38]. These models are able to create realistic images given a text prompt describing just about anything. However, despite the incredible abilities of these systems, discovering the right prompt to generate the exact image user has in mind can be surprisingly challenging. key issue is that all desired aspects of an image must be communicated through text, even those that are difficult or even impossible to convey precisely.
To address this limitation, previous work has introduced methods [9, 14, 18, 30] that tune pretrained models to better control details that a user has in mind. These details are often supplied in the form of reference images along with a new textual prompt [2, 4] or other forms of conditioning [1, 39]. However, these approaches all either rely on fine-tuning with expensive paired data (thus limiting the scope of possible edits) or must undergo costly optimization process to perform the few manipulations they are designed for. While some methods [11, 21, 22, 36] can perform zero-shot editing of an input image using a target caption describing the output, these methods only allow for limited control, often restricted to cructure-preservvii appearance manipulation or uncontrolled image-to- -o-i translation.
By consequence, many simple edits still remain out of reach. For example, how can we move or resize one object in scene without changing anything else? How can we take the appearance of an object in one image and copy it over to another, or combine the layout of one scene with the appearance of second one? How we generate images with certain items having precise shapes specific positions on the canvas? This degree of control has been explored in the past in smaller scale settings [5, 8, 19, 24, 37, 40J, but has not been convineingly demonstrated with modern large-scale diffusion models [26, 32, 38].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

czoissunt aplashing in Beine with the Eircel Tower in background
(a) Original
(b) Swap objects
(c) Enlarge macaron
(d) Replace macaron (c) Copy appearance
(f) Copy seene layout
"A DEL3 photo of seatball and donut falling from the clanda onta ne ighborhoodd
(a) Original
(b) Move donut
(c) Shrink donut
(d) Replace donut
(e) Copy scene appearance
(l) Copy scene layout
Figure I: Self-guidance is method for controllable image generation that guides sampling using the attention and activations of pretrained diffusion model. With seff-guidance we can move or resize objects, or even replace them with items from real images, without changing the rest of the scene (b-d). We can also borrow the appearance of other images or rearrange scenes into new layouts (e-f).
We propose geff-guidance, a zero-shot approach which allows for direct control of the shape, position, and appearance of objects in generated images. Self--uidance leverages the rich representations learned by pretrained text-to-image diffusion models - namely, intermediate activations and attention to steer attributes of entities and interactions between them. These constraints can be user-specified or transferred from other images, and rely only on knowledge internal the diffusion model. Through variety of challenging image manipulations, we demonstrate that simple properties allows for granular, disentangled manipulation the contents of generated images (Figure I). Further, we show that self-guidance can also be used to reconstruct and edit real images.
using only few
Our key contributions are as follows:
We introduce seif-nuidance, which takes advantage of the internal representations of pretrained text-to-image diffusion models to provide disentangled, zero-shot control over the generative process without requiring auxiliary models or supervision.
We find that properties such as the size, location, shape, and appearance of objects can be extracted from these representations and used to meaningfully guide sampling in zero-shot manner.
We demonstrate that this small set of properties, when composed, allows for a wide variety of surprisingly complex image manipulations, including control of relationships between objects and the way modifiers bind to them.
Finally, by reconstructing captioned images using their layout and appearance as computed by self geff-uuidacee, we show that we can extend our method to editing real images.
2 Background
2.1 Diffusion generative models
Diffusion models learn to transform random noise into high-resolution images through a sequential sampling process [12, 33, 35]. This sampling process aims to reverse a fixed iieedeepnneent destructive process that corrupts data by adding noise. The learned component of diffusion model a neural network e, that tries estimate the denoised image, or equivalently the noise &, that was added to create the noisy image 2 agz + O1ea. This network is trained with loss:
L(0)= = B
(()llles - c0(2; v11
(1)
2

---

where is an additional conditioning signal like text, and w(t) is a function weighing the contributions of denoising tasks the training objective, commonly set [12, 15]. A common choice for Eg is a U-Net architecture with self- and cross-attention at multiple resolutions to attend to conditioning text in y [28, 29, 32). Diffusion models are score-based models, where 6g can be seen as an estimate of the score function for the noisy marginal distributions: co(zc) ~ o Va, log p(z) [35].
Given a trained model, we can generate samples given conditioning y by starting from noise 2T AO I), and then alternating between estimating the noise component and updating the noisy image:
& (2(5 v), 2t-1 updatee (=t, G,,,t 1,6+)),
where the update could be based on DDPM [12]. DDIM [34], or another sampling method (see Appendix for details). Unfortunately, naively sampling from conditional diffusion models does not produce high-quality images that correspond well to the conditioning y. Instead, additional techniques are utilized to modify the sampling process by altering the update direction &G.
2.2 Guidance
key capability of diffusion models the ability to adapt outputs after training by guiding the sampling process. From the score-based perspective, we can think of guidance as composing score functions to sample from richer distributions or to introduce conditioning on auxiliary information [7, 17, 35]. In practice, using guidance involves altering the update direction & at each iteration.
Classifier guidance can generate conditional samples from an unconditional model by combining the unconditional score function for p(z)) with a classifier p(y]=z) to generate samples from p( (() oc p(yl=,)
( )p(2)) [7. 35]. To use classifier guidance, one needs access labeled dataset and has to learn a noise- dependee classifier pl ((y(zz) that can be differentiated with respect to the noisy image 2, While sampling, we can incorporate classifier guidance by modifying à
é = c0(2zi 1, y) - s0; V logp(ylzz))
where s is an additional parameter controlling the guidance strength. Classifier guidance moves the sampling process towards images that are more likely according to the classifier [7]. achieving similar effect to truncation in GANs [3], and can also be applied with pretrained classifiers by first denoising the intermediate noisy image (though this requires additional approximations [1]).
In general, we can use any energy function g(2e; t. y) to guide the diffusion sampling process, not just the probabilities from classifier. y could be the approximate energy from another model [17], a similarity score from CLIP model [23], an arbitrary time-independent energy as in universal guidance [1], bounding box penalties on attention [6], or any attributes of the noisy images. We can incorporate this additional guidance alongside classifier-free guidance [13] to obtain high-quality text-to-image samples that also have low energy according to g:
& (1 + s)cg(2itt,)))) y) sce(2,; 0) + vo, V ,.
(4)
where is the classifier-free guidance strength and is an additional guidance weight for g. As with classifier guidance, we scale by a, to convert the score function to a prediction of et. The main contribution of our work is to identify energy functions y useful for controlling properties of objects and interactions between them.
2.3 Where can we find signal for controlling diffusion?
While guidance is a flexible way of controlling the sampling process, energy functions typically used [1, 39] require auxiliary models (adapted to be noise-dependent) well as data annotated with properties we would like to control. Can we circumvent these costs? Recent work [11, 36] has shown that the intermediate outputs of the diffusion U-Net encode valuable information [16, 25] about the structure and content of the generated images. In particular, the self and cross-attention maps (A,,, E I8 x } often encode structural information [11] about object position and shape, while the network activations {V,,, E IR. xR, } allow for maintaining coarse appearance [36] when extracted from appropriate layers. While these editing approaches typically share attention and activations naively between subsequent sampling passes, drastically limiting the scope of possible manipulations, we ask: what if we tried to harness model internals in more nuanced way?
3

---

Pre-trained Diffusion Model
Self-Guidance
"a photo of burrer and an cream cone flcating in occan
fests
none
V centroid
shape
appeacance
Samples
Figure 2: Overview: We leverage representations learned by text-image diffusion models to steer generation with self-guidance. By constraining intermediate activations V and attention interactions At, self-guidance can control properties of entities named in the prompt. For example, we can change the position and shape of the burger, or copy the appearance of ice cream from a source image.
3 Self-guidance
Inspired by the rich representations learned by diffusion models, propose self-guidance, which places constraints on intermediate activations and attention maps to steer the sampling process and control entities named in text prompts (see Fig. 2). These constraints can be user-specified or copied from existing images, and rely only on knowledge internal to the diffusion model.
We identify number of properties useful for meaningfully controlling generated images, de- rived from the set of softmax-normaliin
attention matrices {Ai, E HH xixWWiXKK } and activations (V,,e E R x Wi D. extracted from the standard denoising forward pass eø(2;; object mentioned the text conditioning y token indices attention channel(s) Art.,,.k R xW, x[k] and activations Vi, (extracted timestep from noisy image given text conditioning y) by adding guidance terms to Eqn. 4.
t, y). To control an can manipulate the corresponding
Object position. To represent the position of an object (omitting attention layer index and timestep for conciseness), we find the center of mass of each relevant attention channel:
centroid (k) En.w An.w. w.kk Enaw h Al,ur,
ho An,w.
(5)
We can use this property to guide an object to an absolute target position on the image. For example, to move "burger" to position (0.3, 0.5), we can minimize (0.33, 0.5) - centroid (k) Il We can
also perform a relative transformation, e.g., move "burger" to the right by (0.1, 0.0) by minimizing |centooidggy (k) + (0.1, 0.0) centroid (k) |||-
Object size.
To compute an object's size, we spatially sum its corresponding attention channel:
size (k)
A
(6)
HW
b,a
In practice, we find beneficial to differentiably threshold the attention map Auresh before computing its size, to eliminate the effect of background noise. We do this by taking soft threshold at the midpoint of the per-channel minimum and maximum values (see Appendix for details), As with position, one can guide to an absolute size (e.g. half the canvas) relative one (e.g. 10% larger).
Object shape. For even more granular control than position and size, we can represent the object's exact shape directly through the thresholded attention map itself:
shape(k) = A thresh
This shape can then be guided to match a specified binary mask (either provided by a user or extracted from the attention from another image) with ||target_shapp shape (k) I|1. Note that we can

---

"distast the cokyo with sky"
"a photo futfy cas aitting on museus beach looking at on eil painting of chesse"
photo of racczon in harrel going down vaterfall"
(a) Original
(h) Move up (c) Move down (d) Move left (c) Move right
(f) Shrink
(g) Enlarge
Figure Moving and resizing objects. By only changing the properties of one object (as Eqn. 9), we can move or resize that object without modifying the rest of the image. In these examples, we modify "massive sun", "oil painting
cheese", and "raccoon
barrel", respectively.
apply any arbitrary transformation (scale, rotation, translation) to this shape before using it as a guidance target, which allows us manipulate objects while maintaining their silhouette.
Object appearance. Considering thresholded attention rough proxy for object extent, and spatial activation maps representing local appearance (since they ultimately must be decoded into an unnoised RGB image), we can reach a notion of object-level appearance by combining the two:
appearance(k) En.w shape(k) o
(8)
Σn, shape
4 Composing self-guidance properties
The small set properties introduced in Section 3 can be composed to perform wide range of image manipulations, including many that are intractable through text. We showcase this collection of manipulations and, when possible, compare to prior work that accomplishes similar effects. All experiments were performed Imagen [32], producing 1024 x 1024 samples. For more samples and details on the implementation of self-guidance, please see the Appendix.
Adjusting specific properties. By guiding one property to change and all others to keep their original values, we can modify single objects in isolation (Fig. 3b-3e). For caption C y with words at indices {c:}, in which O = {oj} C C are objects, we can move an object og at time / with:
Fix all other object shapes
LAI Σ
g = wo |0|
Σ
shapee shape; (0)
A
0z0, EO
1=0
Fix appearances
(9)
+ W1
|appearance, .rig appearance,
oEO
Guide translated original shape
LAI
+ 01
Σ
IT
(saape,,...o trring (ox) (ox))),
Where shape and shape extracted from the generation of the initial and edited image, respec- tively. Critically, T lets us define whatever transformation of the H; x W; spatial attention map
5

---

" phaco
"a photo of bear vearing cuit caring his birthday cake out of the fridge in dark kitchen"
(a) Original
(b) New appearances
(c) ControlNet [39] (d) PtP [11]
Figure 4: Sampling new appearances. By guiding object shapes (Eqn. 7) towards reconstruction of given image's s layout (a), we can sample new appearances for given scene (b-d).
want. To move an object, T translates the attention mask the desired amount. We can also resize objects (Fig, 3f-3g) with Eqn. by changing T to up- down-sample shape matrices.
Constraining per-object layout but not appearance finds new "styles" for the same scene (Fig. 4);
Fix all object shapes
LA
(10)
= w/
shape,,,.oo o,t. (0) shape,,
OEO
We can alternatively choose to guide all words, not just nouns or objects, changing summands to Exporec instead of Eofo,E0 See Appendix for further discussion.
Composition between images. We can compose properties across multiple images into a cohesive sample, egg. the layout of an image with the appearance objects another image (Fig. 5):
Copy object shapes from
g = w/0
shape (0) - shape,,,
A
OEO
(11)
Copy object appearance
10
|appearance, (p
We can also borrow only appearances, dropping the first term to sample new arrangements for the same objects, as in the last two columns of Figure
Highlighting the compositionality of self self- guidance terms, we can further inherit the appearance and/or shape of objects from several images and combine them into one (Fig. 6). Say we have / images, where we are interested in keeping a single object ok; from each one. We can collage these objects "in-place" i.e. maintaining their shape, size, position, and appearance straightforwardly:
Copy each object's shape, position, size
Wy j
shapee (ok,) - s
(okk))
A
(12)
Copy each object's 's appearance
W1 (oppearance.,(ox.) (ppearance,(o+)ll
We can also take only the appearances of the objects from these images and copy the layout from another image, useful if object positions in the J images not mutually compatible (Fig. 6f).

---

"a photo of suiscase, bowling ball, and phone vashed beach after shapwreck"
Appearance
Layout +
No layout conditioning
Figure 5: Mix-and-match. By guiding samples take object shapes from image and appearance from another (Eqn. 11), we can rearrange images into layouts from other scenes. Input images are along the diagonal. We can also sample new layouts of scene by only guiding appearance (right).
photo picnic
(a) Take blanket
Take
(c) Take
Result
(c) Target layout (f) Final result
"a cop-docn photo of tea kettle, boot of fruit, and of patcha"
(a) Take matcha
(b) Take kettle
(c) Take fruit
(d) Result
(c) + Target layout (f) Final result
"a photo of dog vearing knit sveater and baseball cap drinking cocktail"
(a) Take sweater (b) Take cocktail
(c) Take cap
(d) Result*
(e) + Target layout (f) Final result
Figure 6: Compositional generation. new scene (d) can be created by collaging objects from multiple images (Eqn. 12). Alternatively egg. if objects cannot be combined at their original locations due incompatiiilities these images' layouts (*as in the bottom row) we can borrow only their appearance, and specify layout with new image (c) to produce a composition (f) (Eqn. 19).
7

---

puppee
outZIE
supezman floating
pouring flosting
Ours
DreamBooth
Figure 7: Appearance transfer from real images. By guiding the appearance of a generated object match that of one real image (outlined) in Eqn. 13, we can create scenes depicting an object from real life, similar to DreamBooth [30], but without any fine-tuning and only using one image.
"a photo hot fries,
solid backkgroudd
(a) Real image
(b) Reconstruct
(c) Swap w. fries
(d) Width +
(e) Width 4. height
f
(l) Restyle
photo eclair shot wagresae"
(g) Real image
(h) Reconstruct
(i) Move
(j) Width +
(k) Width, height t
(l) Restyle
Figure 8: Real image editing, Our method enables the spatial manipulation of objects (shown in Figure for generated images) for real images well.
Editing with real images. Our approach is not limited to only images generated by a model, whose internals we have access to by definition. By running noised versions of a (captioned) existing image through denoiser one for each forward process timestep intermediates that can be treated if came from a reverse sampling process (see Appendix for more details). In Fig. 8, we show that, by guiding shape and appearance for all tokens, we generate faithful reconstructions real images. More importantly, we can manipulate these real images just as we can generated ones, successfully controlling properties such as appearance, position, or size. can also transfer the appearance of an object of interest into new contexts (Fig. 7), from only one source image, and without any fine-tuning:
extract set of
Copy object appearance
g = wi |appearance, priig (Okwc)) appearance, (ok) ||1
(13)
Attributes and interactions. So far have focused only the manipulation of objects, but we can apply our method to any concept in the image, as long as it appears in the caption. We demonstrate manipulation of verbs and adjectives Fig. and show an example where certain self-guidance constraints can help in enforcing attribute binding in the generation process.
5 Discussion
We introduce a method for guiding the diffusion sampling process to satisfy properties derived from the attention maps and activations within the denoising model itself. While propose number of

---

cat monkoyy laughing on
phoco xecey
"green hat, book, yellow shoee, jacket"
Move laughing to the right
Change messy location
Move red to jacket, yellow to shoes
(a) Original
(b) Modified
(c) At (0.3, 0.6) (d) At (0.8,0.8)
(c) Original
(f) Fixed
Figure Manipulating non-objects. The properties of any word the input prompt can be manipulated, not only nouns. Here, we show examples of relocating adjectives and verbs. The last
example shows a case in which additional self-guidance can correct improper attribute binding.
plotto aquirrel trying to carch lime xid-
"a potato aitting couch with boul popcorn vatching football TV"
"a picture of
All Lokens
oskee
Appearance features leak layout (a) Unguided (b) "lime" guided
Multi-token layout leaks appearance
Interacting objects entangled
(c) Real image
(d) Layout guided
(e) Original
(I) Move potato
Figure 10: Limitations. Setting high guidance weights for appearance terms tends introduce unwanted leakage of object position (a-b). Similarly, while heavily guiding the shape of one word simply matches that object's layout as expected, high guidance on the shapes of all tokens results in leak of appearance information (c-d). Finally, in some cases, objects are entangled in attention space, making it difficult control them independently (e-f).
such properties, many more certainly exist, as do alternative formulations of those presented in this paper. Among the proposed collection of properties, few limitations stand out.
The reliance on cross-attention maps imposes restrictions by construction, precluding control over any object that is not described in the conditioning text prompt and hindering fully disentangled control between interacting objects due to correlations attention maps (Fig. 10e-10r), Selectively applying attention guidance at certain layers timesteps may result in more effective disentanglement.
Our experiments also show that appearance features often contain undesirable information about spatial layout (Fig. 10a-10b), perhaps since the model has access to positional information in its architecture. The reverse also sometimes true: guiding the shape of multiple tokens occasionally betrays the appearance of object (Fig. 10c- 10d), implying that hidden high-frequency patterns arising from interaction between attention channels may be used to encode appearance. These findings suggest that our method could serve window into the inner workings of diffusion models and provide valuable experimental evidence to inform future research.
Broader impact
The use-cases showcased in this paper, while transformative for creative uses, carry the risk of producing harmful content that can negatively impact society. In particular, self- eelf-guidance allows for level of control over the generation process that might enable potentially harmful image manipulations, such pulling in the appearance or layout from real images into arbitrary generated content (e.g., as Fig. 7). One such dangerous manipulation might be the injection a public figure into an image containing illicit activity. In our experiments, we mitigate this risk by deliberately refraining from generating images containing humans. Additional safeguards against these risks include methods for embedded watermarking [20] and automated systems for safe filtering of generated imagery.
Acknowledgements
We thank Oliver Wang, Jason Baldridge, Lucy Chai, and Minyoung Huh for their helpful comments. Dave is supported by the PD Soros Fellowship. Dave and Allan conducted part of this research at Google, with additional funding provided by DARPA MCS and ONR MURI.

---

References
[1] Arpit Bansal, Mong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. arXiv preprint arXiv:2302.07122, 2023.
[2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 707-723. Springer, 2022.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, hts//809.11096 2018.
[4] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
[5] Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in gans. arXiv preprint arXiv:2103.10426. 2021.
[6] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03375. 2023.
[7] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, 2021.
[8] Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. Blobgan: Spatially disentangled scene representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 616-635. Springer, 2022.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.011 01618, 2022.
[10] Alexandros Graikos, Nikolay Malkin, Nehojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv:2206.09012 2022.
[11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint urXiv:2208.01626, 2022.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS. 2020.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.125595 2022.
[14] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09271 2022.
[15] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurIPS, 2021.
[16] Mingi Kwon, Jaescok Jeong, and Youngjung Uh. Diffusson models already have semantic latent space. arXiv preprint arXiv:2210. 10960, 2022.
[17] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, 2022.
[18] Vivian Liu and Lydia B Chilton. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the 2022 CHI Conference on Human Factors Computing Systems, pages 1-23, 2022.
10

---

[19] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124, PMIR, 2019.
[20] Xiyang Luo, Michael Goebel, Elnaz Barshan, and Feng Yang. Leca: A learned approach for efficient cover-agnostic watermarking. arXiv preprint arXiv:2206. 10813, 2022.
[21] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Enmon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[22] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 2022.
[23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Iya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
[24] Taesung Park, Jun-Yan Zhu, Oliver Wang. Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 33:7 13:1199-7211, 2020.
[25] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10629, 2022.
[26] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
[27] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, 2021.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/ 550.04497, 2015.
[30] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. /2242, 2022.
[31] Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. SIGGRAPH, 2022.
[32] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho. David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv:2205.
2022.
[33] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. ICML, 2015.
[34] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. CORR, abs/2010.02502, 2020.
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
11

---

[36] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-pla diffusion features for text-driven image-to- tongge translation. arXiv preprint arXiv:2211.12572 2022.
[37] Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprini arXiv:2107.07905. 2021.
[38] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv:2206. 10789, 2022.
[39] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[40] Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, and Qifeng Chen. Region-based semantic factorization in gans. arXiv preprint Sttiv:::202.096499 2022.
12

---

Appendix
A. Implementation details
We apply our self-guidance term following best practices for classifier-free guidance on Imagen [32]. Specifically, where N is the number of DDPM steps, we take the first 3N and the last N without. The remaining 25.V 32
16 steps with self-guidance
32 steps are alternated between using self- guidance and not using it. We use N = 1024 steps. Our method works with 256 and 512 steps well, though self-guidance weights occasionally require adjustment. We set v = 7500 in Eqn. 4 as an overall scale for gradients of the functions 9 defined below quite small (often in the range of 10-7 to 10-6, so such a high weight is needed to induce changes.
find that the magnitude of per-pixel gradients is
We apply centroid, size, and shape terms on all cross-attention interactions in the model we use. In total, there are 36 these, across the encoder, bottleneck, and decoder, at 8 x 8, 16 x 16, and 32 32 resolutions. We apply the appearance term using the activations of the penultimate layer the decoder (two layers before the prediction readout) and the final cross-attention operation. We experimented with features from other parts of the U-Net denoiser, namely early in the encoder before entanglement),
positional information can propagate through the image (to prevent appearance- but found these to work significantly worse. To avoid degenerate solutions, we apply stop- goo-ggradent to the attention in the appearant ance term so only information about activations is back- propagated. We take the mean spatially of all shape terms and across activation dimensions for all appearance terms, which we omit in all equations for conciseness.
Attention mask binarization. In practice, it is beneficial to differentiably binarize the attention map (with sharpness controlled by s) before computing its size or utilizing its shape, to eliminate do
the effect of background noise (this is empirically less important when guiding centroids, not binarize in that case). We do this by taking soft threshold at the midpoint of the per-channel minimum and maximum values. More specifically, we apply shifted sigmoid on the attention normalized to have minimum O and maximum 1, followed by another such normalization to ensure the high value I and the low after applying the sigmoid. We use 10 and redefine Eqn. 6.
X ming, (X)
normalize(X)
(14)
maxh, (X) mins, (X)
thresh normaliza ize cermallzessigmoiie (s (Aormalize(A) 0.5)))
(15)
size (k) HW
Σ
A,u,k
(16)
b,w
"a of and bot outdoore"
"a photo of at oak tree and pineapple outside an arctic igloo"
"a photo of an ovl and pig running at the racetrack"
(a) Original
(b) Edited
Figure 11: Moving objects. Non-cherry-picked results for moving objects in scenes using Eqn. 9. We move onion down and to the right, pineapple to the right, and owl up and pig down, respectively. All scenes use weights wi 1.5, w = 0.25, and W22 = 2.
13

---

photo kangaroo punching bag
23TH
photo
walkinng
eports
ghoto of boxabox on canol cear
pead
(a) Original
(b) Edited
Resizing objects. Non-cherry-pice piccked results for resizing objects in scenes using Eqn. with T specified to up- down-sample attention maps. We reduce the punching bag's height 0.5 and enlarge chi ckin 2.5x and boombox 2x. All scenes use Wi = 2, w1 0.25, and W2 = 3.
Figure 12:
B. Using self-guidance
Maximizing consistency. general, we find that sharing the same sequence of noise in the DDPM process between image and its edited version is not necessary to maintain high levels of consistency, but can help extreme precision desired. under transformations such resizing and repositioning is more effective if applying transformation T to the original shape, rather than expressing the same change through centroid and size.
find that maintaining object silhouettes
Guiding "background" words. To keep all objects of the scene fixed but one (Fig. 3), one can either guide all other tokens in the prompt (including "a photo of" and other abstract terms) to keep their shape, or only select the other salient objects and hold those fixed. In general, since abstract words are often used for message passing and have attention patterns that are correlated with the layout of the scene, we prefer not guide their layouts maximize compositionality.
Mitigating appearance-layout entanglement. When words or concepts span multiple tokens, we can mean-pooling attention maps across these tokens before processing them, though do not find this to improve results. We also find that corrupting target shapes with Gaussian noise helps mitigate this effect, providing some evidence for this hypothesis.
Moving objects. We wi E [0.5, 2), wi E [0.03, 0.3) u2 [0.5, 5) in Eqn. 9. Alternatively, can express or new location through its centroid, adding term to keep size fixed:
object shapes
9 = Wo
|sshape.r.orig shape,
0
object appearances
+ W1
aappearance,.oo .aig (o) appearance, (0)l1
(17)
Fix 'sizc
LA
+
siize,,..oii())) - size),((x)]l
A
+ W33
|target_centtoidd centrrodddd
[
Change position
14

---

photo picking
ploto
capybara rearing robe citting the tireplace"
phoce drinking
asyle
(a) Original
(b) Edited
Figure 13: Creating new appearances for scenes. Non-cherry-picker results sampling different "styles" of appearances given the same layout, using Eqn. 10. We use wi 0.7, 0.3, and 0.3
respectively for each result, to preserve greater structure in the background of the first picture.
Where target_centee centroid be computed specific centroidung if desired, or selected to an absolute value on the canvas (repeated across all timesteps). We generally weights wo E [0.5, 2]. wn E [0.03, 0.3), w0 [0.5, wy E [1,3].
shfited version of the tinestep-and-attentiorn
Resizing objects. We can follow Eqn. resize objects as well, by setting to upsample or downsample the original mask. We can similarly use Eqn. 17, omitting the final term and setting the target size to a desired value, either computed as a function of sizeorig (ok) or provided as an absolute proportion of pixels on the canvas that the object should cover. We use the same weight range for weights except we set Wy [1, Wy = for Eqn. 17.
Sampling new appearances. We set [0.1, 1] in Eqn. 10. Generally, higher values lead to extremely precise layout preservation the expense of diversity in appearance.
Sampling new layouts. Just as we can find new appearances for a scene of given layout, we can perform the opposite operation, finding new layouts for scenes where objects have given appearance:
Fix all appearances
(18)
= wD O
|appearance, tunig (o) appearance,(0)) (o)
oEO
We almost always wo [0.05, 0.25).
Collaging objects in-place. Eqn. 12 can be easily generalized to more than one object per image (adding another sum across all objects) to the case where prompts vary between images (mapping from ky to the corresponding indices in the new image). We set wy [0.5, 1). w1 E [0.05, 0.3].
Collaging objects with new layout. As shown in Fig. 6f, we can also collage objects into new layout specified by target image 1, in addition the images specifying object appearance:
Copy all object shapes
A
W0
|sapee i J4 (0ks4) (ox)));,(x)))
(19)
Cupy each object's appearance
w1
apppearance,, appearance,(o:)]
As in Eqn. 12, set wo [0.5, 1). [0.05, 0.3].
15

---

birtaday party
"a photo cleate, bright soccer ball, and come"
toy caz, pillov rag"
(a) Original
(b) Edited
Figure 14: Creating new layouts for scenes. Non-cherry -iccked results sampling new layouts for the same scenes, using Eqn. 18. We use wl = 0.07, 0.07, and 0.2 respectively.
"a BSLR of backpack the
DSLA photo
backsaack
vater"
"a
(a) Original
(b) Object in new contexts
Figure 15: Appearance transfer from real images. Non-cherry picked results sampling new images with given object's appearance specified by a real images, as in Eqn. 13. We use wp 0.15.
Transferring object appearances new layouts. Nothing requires the indices (or fact, the objects those indices refer to) to be the same in the image being generated and the original image being used as source, as long there is a mapping specified between the indices the old and new images which should correspond. Call this mapping m. We can then take the appearance of an object оą a source image and transfer an image with any new prompt as follows, as specified in Eqn. 13 (with typical weights [0.01, 0.1]):
Cupy object appearance
g wi appearannce,orrr orig (ok) appearance, (m((x)) ||]
(20)
Merging layout and appearance. We use wo E [1,2] and [0.1,0.3 in Eqn. 11.
Editing with real images. Importantly, our method is not limited editing generated images to whose internals it has access by definition. We find that can also mcaningfully guide generation
16

---

photo learing rigoraan
"a deir photo of beapet floating in the sea'
(a) Original
(b) Ours
(b) Random samples without self- self-guidance
Figure 16: Ablating appearance transfer from real images. To verify the efficacy of our approach, we compare our results from Fig. 7 in the paper to random samples from the same prompt without apperance transfer. We can see that appearance of objects varies significantly without self-guidance.
using the attention and activations extracted from set of forward-proces. denoisings of real image (given a caption) to "approximate" the reverse process, despite any mismatch in distributions one might imagine. Concretely, we generate T' corrupted versions of real image I, {azz + oye J where e, (O, 1). We then extract the attention A, and activations V, from the denoising network each of these timesteps in parallel and concatenate them into length-T sequence. We treat this sequence identically to sequence of T internals given by subsequent sampling steps, and can thus transfer the appearance of objects from real images, output images that look like real images with moved or resized objects, and so on.
transfer appearance "A photo of Chow Chow..." and "A DSLR photo of teapot..." While our method works on less specific descriptions as well, is not as reliable when object appearance is more out-of-distributtinn For context, we show unguided samples under the prompts from Fig. in Fig. 16, which still deviate significantly from the desired appearance, showing the efficacy our approach. A weakness of our simple approach is that it has no constraints on the shape of the generated objects, which we leave to future work.
In Fig. 7, the prompts we
Weight selection heuristics. We find weights that work well to remain more or less consistent across different images given an edit, but ideal weights do vary somewhat (within predictable ranges) between different combinations of terms. Our heuristies for weight selection per term are: the more weights there are, the higher per-term weights can be without causing artifacts (and indeed, need to be, to provide ample contribution to the final result); appearance terms should have weights orders of magnitude lower than layout terms; layout summary statistics (centroid and size) should have slightly lower weights than terms on the per-pixel shape; total weight terms should not add up to more than ~ 5 to avoid artifacts.
C. Additional results
We show further non-cherry-picked results for the edits we show in the main paper. Our general protocol consists of selecting an interesting prompt manually, verifying that our model creates compelling samples aligning with this prompt without self- -ufidnu,,,ee,, beginning with the typical weights we use for an edit, and trying around 3-5 other weight configurations to find the one that works best for the prompt - in most cases. this is the starting set of weights. Then, we use the first 8 images we generate, without further filtering. We generate all results with different seeds to showcase the strength of guidance even without shared DDPM noise. We show more results for moving (Fig. 11) and resizing (Fig. 12) objects, sampling new appearances for given layouts (Fig. 13) as well as new layouts for a given set of objects (Fig. 14), and transferring the appearance of real objects into new contexts (Fig. 15). We also include an ablation hyperparameter values (Fig. 17) as well as preliminary results an implementation of self-guidance on an open-source diffusion model in Fig. 18.
17

---

Original: phott ot recaree unne vaible"
Oetginal: "s plate reeazall and creat callng theckaad exzo neightettedd
Bdit odagee sunuuce
Bdlk: save melllll
Wol Sagee
0.4
1.4
5.2
64
0
1.5
32
0.56
0.34
0.32 0.1
K,40.41
20 116 0.33 66
Figure 17: Hyperparameter sweeps. We show results for two edits (moving and resizing objects) using Eqn. in the Supp. Mat., for different values of weights the three edit terms, holding the other terms to the value the middle column. Please zoom in to view in more detail. Reasonable values in the middle columns (within the expected range) lead overall successful image manipulation. Very large hyperparameter values cause visual artifacts to appear (by moving sampling off-manifold) while still tending to perform the edit successfully, while extremely small values often fail conduct the edit, inducing artifacts resulting from a "half-executed manipulation.
pitcher picnic
playing volleyball
watermelon controid (0.3, 0.7)
watermelon centroid (0.7,0 0.7))
volleyball centroid (0.85, 0,15)
volleyball centroid (0.2. 0.15)
"a tropical
Graneise"
foussain pen, whiskey desk"
centroid (0.3,
frog cencroid (0.65, 0.5)
controld (0.35, 0.35)
globe centrid (0.9, 0.9)
geese flynng through the sky top
chasing tomorisedhell
0.05
Figure 18: Self-guidance on Stable Diffusion XL.. To highlight the generality of our approach, we demonstrate preliminary results for controllable generation on popular latent-space text-to-image diffusion model, using 100 DDPM steps (applying self-gum gufdance from step 10 to step 90). We get best results only guiding attention in the second decoder block of the denoiser model.
18


Reviews:
[{"review_id": "I061ziReaH", "reviewer": "Reviewer_EBx5", "summary": "This paper proposes a method for image editing using pretrained text-to-image diffusion models. The method guides the sampling process with energy functions that are added similarly to classifier guidance. These energy functions are computed as the difference between some object's property and the target state we want to change it to. A property is either an object's location, size, shape, or appearance. These properties are represented by the cross-attention maps between the object's word token and the image, which has previously been shown to produce meaningful masks. Multiple energy functions can be composed to achieve composite edits. Experiments qualitatively demonstrate the appeal of the method.", "questions": "What pretrained diffusion model is used in the experiments? Does this method perform differently for different diffusion models (Stable Diffusion vs. eDiff-I vs. Imagen vs. etc.)? I'm asking this because different diffusion models incorporate text conditioning differently (both architecturally and during training).\n\nHow sensitive is the method w.r.t. hyperparameters like the attention layer, time step, and guidance strength? Is it necessary to tune the hyperparameters for every image individually? Maybe even for every energy function individually?", "limitations": "Many limitations regarding the method and potential negative societal impacts are discussed. Some that might be reasonable to also mention are listed above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is practical since it does not require retraining while still achieving impressive results (as demonstrated qualitatively with some samples).\n\nThe paper proposes a simple method with a clear and concise presentation.\n\nThe addressed problem (image editing using pretrained diffusion models) is of interest to both academics and practitioners. ", "weaknesses": "Quantitative evaluations of the method (for example with user studies that rank/compare different editing results) would make this work more insightful. Currently, it is unclear whether the shown examples are representative of what to expect when using this method and where the limitations lie. A potential limitation that could be exposed through quantitative studies is the number of energy functions that can be composed at once. Editing real images is possible as shown in Figure 7, but it isn't clear how well it works when compared to generated images. Furthermore, from some examples (e.g., Figure 8 a-b) it looks like the non-edited parts of the image also change and do not remain consistent with the original image, as is usually desirable in image editing settings.\n\nThere are missing details that could help in reproducing and better understanding the results. See the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and thoughtful comments. \n\n**Open-source Stable Diffusion implementation:** To facilitate an even better understanding of the limitations of self-guidance, as well as address questions about whether shown examples are representative of the method’s abilities and whether our method generalizes beyond Imagen, we are releasing an implementation of self-guidance on Stable Diffusion XL. We have also attached examples of controlled generation using self-guidance on SD-XL in Fig. 2 of the PDF in the global comment.\n\n**Limitations of self-guidance:** We agree that strong numerical evaluation of controllable image generation approaches remains an open problem. We qualitatively show that our approach in quality is comparable to previous work where there is overlap in functionality (e.g. Figs. 3, 6) though it requires no fine-tuning. Additionally, our method enables many manipulations that are not possible with previous work – and we include a representative demonstration of the main shortcomings of our method in  Fig. 9. We also show that many energy functions can be composed (Fig. 4) with terms from multiple source images on appearance and layout – we find that as long as the specified constraints do not contradict each other, the effectiveness of self-guidance does not decay as more terms are added.\n\nWe also show additional non-cherry-picked examples of image manipulation in the Supplementary Material, highlighting the diversity of outputs the method is capable of producing.\n\nAs for editing real images, we do find a slight decrease in faithfulness to the original, which seems to stem from the limitations of the simple appearance term in our reconstruction methodology (an average-pooled per-token vector). \n\nWe thank the reviewer also for pointing out slight inconsistencies in the background of the edit in Figure 8 (a,b). In this case, a higher-quality result can be obtained by sharing DDPM noise between the two images (as described in Supp. Mat L27-29), a technique we did not apply to results in that figure .\n\n**Sensitivity to hyper-parameters:** We use Imagen in all experiments shown in the paper, but our method only makes the assumption that a cross-attention interaction between text and image tokens exists in the architecture (which is the case for all current SOTA text-to-image systems). We guide all attention layers and timesteps across all edits, without per-edit tuning. Each energy function has its own “default value” weight that works well for it, and this value is not dependent on the prompt or image at hand. When composing many energy functions, one may wish to slightly increase or decrease the strength of one term when compared to others, but we find that the range of per-term weights across images is small and that any value within the range works reliably. We find hyperparameter ranges by running an ad-hoc binary search once for each self-guidance term (e.g. centroid, size, …): too-large values cause visible artifacts in image quality while small values fail to cause the edit to take place. We include details on these hyperparameters in Supp. Mat. (e.g. line 53, 58) as well as a visual demonstration of their effects in the one-page PDF attached to the global response to all reviewers.\n"}, {"review_id": "v2KcFCe9CP", "reviewer": "Reviewer_jH2E", "summary": "This paper introduces diffusion self-guidance, an inference-time technique for controllable image generation using pre-trained text-to-image diffusion models. The key finding is that internal representations of the denoiser network carry meaningful information about the scene, and one can build custom energy functions around these representations to align image generation with user-defined scene properties, including object location, size, shape and appearance. Further, an appealing property of self-guidance is that the energy functions can be flexibly composed to support the simultaneous manipulation of multiple image attributes. Extensive experiments are conducted to demonstrate the effectiveness of self-guidance across a broad spectrum of image manipulation tasks.", "questions": "Please find my comments in the section above.", "limitations": "This paper includes a discussion on the limitations and potential societal impact of the proposed method.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed method enables controllable image generation / manipulation using existing text-to-image diffusion model checkpoints without costly fine-tuning. As a training-free method, it achieves controllability by steering the generation towards matching the desired internal representations at inference time. This guidance-based design is conceptually simple, easy to implement, computationally efficient and highly flexible.\n\n- The method represents object properties using aggregated statistics of denoiser representations. It thus reveals the internal working of the generative process. To this end, the impact of the paper goes beyond controllable image generation / manipulation.\n\n- Self-guidance enables several edits that are not possible with concurrent methods. These include change in object location, size and modification of scene layout. In the meantime, self-guidance allows the composition of multiple edits in a single generation pass. This is another unique property that is rarely seen in previous and concurrent works.\n\n- Finally, self-guidance offers compelling editing capabilities and supports a wide range of image manipulation needs. Among them are the mixing of objects and layout from multiple source images, transfer of attributes defined by non-object words, and the editing of real images.", "weaknesses": "I do not have major concerns about the proposed method. Some minor concerns are as follows.\n\n- The main text presents self-guidance in its generic form and provides the ingredients for defining a specific energy function. However, all concrete examples are left to the supplement. I understand that space is limited and the authors want to make room for the results (which are indeed compelling). However, it is awkward to alternate between the main text and the supplement in order to decipher how those edits are actually achieved. One possibility is to organize all instantiations of self-guidance in a big table and put them in the main text for the ease of understanding.\n\n- I did not find ablation experiments on hyper-parameters such as the number of inference steps, guidance strength and the range of steps to apply guidance. In practice, the editing quality is often quite sensitive to these hyper-parameters. While their choices are briefly discussed in the supplement, some visual examples will help visualize their effects.\n\n- The experiments are conducted using Imagen, whose design is substantially different from the publicly available Stable Diffusion (cascaded vs. latent diffusion). It is a well-known fact that an editing method may not work equally well on different diffusion models. Given that most players in the space of controllable image generation rely on Stable Diffusion, I would encourage the authors to showcase a few editing examples using Stable Diffusion in order to calibrate the performance of self-guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and review our paper!\n\n**Paper layout:** Thanks for the feedback on this. We entirely agree and have restructured the text so that the energy functions are in the main paper rather than the supplementary material.\n\n**Sensitivity to hyper-parameters:** We always apply self-guidance on all inference steps, and find that our method works on a wide range of steps – we experimented with 128, 256, 512, and 1024. Coarsely, we find that applying self-guidance only on early steps allows for control of layout but not appearance, and only on later steps allows for control of some appearance but not layout. That being said, we agree with the review that visual demonstrations of these effects are valuable to provide a better understanding of the method’s strengths and weaknesses, and we have included this ablation figure in the global comment (and will also incorporate it into the final manuscript). Thank you for the suggestion. Please also see response to reviewer EBx5 for more discussion. \n\n**Open-source Stable Diffusion implementation:** We agree that showcasing the method’s ability to generalize beyond one specific text-to-image diffusion model is valuable and are releasing an implementation on Stable Diffusion XL. Please see the PDF in the global comment for examples of self-guidance applied to Stable Diffusion XL."}, {"review_id": "VD7FzFKBYn", "reviewer": "Reviewer_hErC", "summary": "Large-scale text-to-image generative methods have demonstrated impressive performance given a detailed text prompt. However, many aspects of an image are hard or impossible to specify with texts. The authors proposed self-guidance to guide the generation of diffusion models. Experimental results demonstrated that object size, location, and appearance of objects can be controlled from normalized attention matrices and feature activations.", "questions": "1. To see how entangled the representations are, can the authors generate a video or a dense sampling of intermediate images to interpolate the “control space”? For instance, changing the layout of the objects in Figure 5 over 16 frames, or changing the styles in Figure 4 over 16 frames? In this way we could see: (i) if a continuous control corresponds to smooth and continuous changes in the image space, and (ii) how the other objects are influenced by the controlled changes made to the target object.", "limitations": "1. The input space of the proposed control is still limited by texts: see weakness 1.\n2. The proposed approach is haunted by entangled representation: see weakness 2 and 3.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation and background is explained clearly. Arguments and claims are supported by references or experimental results.\n2. The proposed approach is simple and effective. The presented empirical results on multiple tasks demonstrated the effectiveness of the approach. The authors claimed that all results are non-cherry-picked, which is quite impressive.\n3. Adding geometrical/compositional control to the image generative models is an interesting and important research topic. The authors proposed a simple and smart method to control existing generative models with diffusion guidance.\n4. Multiple tasks were considered, including adjusting individual properties, compositional generation, and real image editing. This proposed approach showed potentials to a range of real-world applications.", "weaknesses": "1. From the abstract, the authors aimed to add more control over the image generation beyond text prompts. However, the proposed approach is not a good solution in this regard – it is strongly based on the tokens in the text prompt. There are two limitations:\n   * Can we control fine-grained objects/parts not naturally present in the text prompt? For example, control the dog mouth when generating “a photo of a dog wearing a baseball cap”? All current results are on the object-level with no finer control.\n   * The proposed approach is limited to objects/parts easily describable by texts. This is a disadvantage compared to the keypoint inputs from DragGAN. For instance, annotating a keypoint would be more straightforward than inputting “left-front leg of the dog”, for both the user and the model.\n2. Based on the framework, designing more control seems difficult, e.g., 3D viewpoint of objects. Compared to parallel works, ControlNet [2] finetuned the LDM model with 3D representations and DragGAN [1] used finer keypoint inputs to accomplish this.\n3. The proposed method is built on existing large-scale image diffusion models. It heavily relies on a good representation learned by these trained models. The aforementioned limitations are mostly due to the latent representations having limited feature disentanglement or 3D awareness. The proposed approach doesn’t seem to have any easy fixes to address these limitations.\n\nReferences:\n1. X. Pan et al. Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold.\n2. L. Zhang et al. Adding conditional control to text-to-image diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback on our work.\n\nWe respectfully disagree that self-guidance is not a “good solution” to “add more control over image generation beyond text prompts”. The edits that we perform, such as moving or resizing objects, are very challenging (if not impossible) to affect through text, especially if consistency with a source image is desired. Indeed, no previous method has shown this functionality. We acknowledge in the manuscript that self-guidance as we present it requires entities to be mentioned in the text prompt, but please also see discussion under “Limitation to entities mentioned in prompt” in the response to reviewer qibF. \n\nSelf-guidance enables a wide variety of image manipulations that – though they rely on text-image attention – *are not themselves feasible just through prompting with text*. Concurrent work such as DragGAN has been implemented in diffusion models already, in fact, building on self-guidance [1], highlighting the strength and versatility of our approach. 3D viewpoint control is an interesting future direction, but we restrict our attention to 2D for this work. The fact that self-guidance relies on the strong representations learned by text-to-image models is a feature, not a bug – we show that many complex image manipulations require *no additional fine-tuning or supervision or auxiliary models* (which ControlNet does). We wish to highlight that many of the manipulations enabled by self-guidance have not been shown at all in the literature for diffusion models previously.\n\n[1] DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models, Mou et al., 2023\n"}, {"review_id": "Uowi0YtaYk", "reviewer": "Reviewer_qibF", "summary": "The paper introduces a new method for detailed and versatile image editing and controlled image synthesis with large-scale text-to-image diffusion models. In particular, the authors propose self-guidance: Self-guidance uses the internal representations of the denoiser neural network of diffusion models for guidance. The work finds that the attention and feature maps of the denoiser network encode properties like objects' size, shape, position and appearance in an interpretable way. This is, some heuristics on the internal attention/feature maps can be defined that represent these properties well and then we can do guidance, leveraging these internal representations (note that this works only for objects and words that were mentioned in the text prompts driving generation). Hence, given an image (either generated by the diffusion model itself or a real one, reconstructed by the diffusion model), the novel self-guidance technique enables editing operations such as freely modifying objects' size, shape, position and appearance, without introducing noticeable artifacts. Importantly, this does not require any additional training or use of auxiliary models, and no additional paired data for training, which makes this a very powerful technique. The authors convincingly validate their proposed editing method mostly qualitatively by showing lots of interesting editing examples.", "questions": "I also have a variety of questions that did not significantly impact the rating of the paper:\n- I believe the paper intrinsically relies on a typical U-Net architecture for the denoiser network that gives rise to the useful attention and feature maps. There are various recent works that use pure vision transformer denoiser networks or advanced architectures like recurrent interface networks. Can the approach be extended to these architectures, too? I would be interested in the authors' thoughts on this.\n- In a similar spirit, do the authors believe that similar techniques can also be used in diffusion models that model video, 3D, audio, graph/molecule data, etc.?\n- While centroid, size and shape leverage all cross attention maps, the appearance term uses only the activations of the penultimate layer in the decoder as well as the final cross-attention operation. Why is that? Can the authors provide more intuitions for the qualitative differences in how the attention/activations are used in the different operations?\n- The paper entirely leverages DDPM-style sampling. Does the method also work with deterministic probability flow/DDIM sampling? Moreover, when using given real images, could there be an alternative way of editing when probability flow ODE/DDIM is used to deterministically encode a given image and then re-generate again while introducing the desired edits? I would be curious about the authors' perspective on this.\n\nVery Minor nitpick: \n- Please properly introduce the data $x$ as well as the noise schedule $\\alpha_t$, $\\sigma_t$ around equation 1.\n- I would suggest to properly introduce and define the indices in Line 155 and below, this is, what exactly is $i$, etc. to avoid any potential confusion.", "limitations": "Limitations and potential negative societal impact have been appropriately discussed and are well addressed. There are no concerns in that regard.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Generally, the main strength of the paper is the introduction of the novel self-guidance technique, which it then uses for detailed and advanced image editing and controllable image synthesis. To the best of my knowledge no previous image editing methods with text-to-image diffusion models reach this level of fine control with respect to objects' shape, size, position, appearance, layout, and composition, while introducing almost no artifacts.\n\n**Clarity:** The paper is very well written and easy to follow and understand. It is very clear.\n\n**Originality:** The main idea of self-guidance leveraging the model's internal representations is novel and original. That said, it is closely related to, and probably inspired by, existing technique such as Prompt-to-prompt editing (Hertz et al. \"Prompt-to-Prompt Image Editing with Cross Attention Control\") and paint-with-words (Balaji et al. \"eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers\").\n\n**Significance:** Controllable image generation and image editing with large-scale text-to-image diffusion is an important direction with many relevant applications. The paper proposes a very powerful and versatile method with little requirements, which makes this a significant contribution.\n\n**Quality:** The overall quality of the paper is high. It is clear, well written, all claims are supported, limitations and societal impacts are appropriately discussed, background is provided, and related work is thoroughly discussed.", "weaknesses": "While the paper has few major issues, there are some concerns:\n- Editing and control is limited with respect to objects (and words more generally) that appear in the text prompt. This also means that given real images require an appropriate caption to be editable. Note that this limitation is acknowledged by the authors.\n- The method comes with a ton of hyperparameters: The guidance weights for the different edits need to be chosen manually and it seems like some trial and error can be necessary. Also, more detailed ablations on this would be helpful, like showing different guidance weights for same image/seed/prompt/edit, etc.\n- When performing a simple edit like moving a single object, we actually require a lot more guidance terms for keeping all the other objects in place and preserving their appearance. This means that even edits that may appear simple are actually quite complicated and require guidance weights for all different terms.\n- The way the different edits and controls are defined is largely heuristic (how size, shape, appearance, etc. is calculated from the attention/feature maps). It seems to work well, but in theory it would be great if there was a more principled approach to this.  \n- The way the cross-attention maps are leveraged in self-guidance is related to the \"paint-with-words\" technique introduced in eDiff-I (Balaji et al. \"eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers\"). I believe this should be acknowledged and cited.\n- The approach is validated only in pixel-space text-to-image diffusion models. Can the same method also work in latent diffusion models? The most popular, and publicly available, large-scale text-to-image model is Stable Diffusion. It would be great to see the method also applied there, if possible.\n\nIn conclusion, there are concerns, weaknesses and routes for further improving the paper. However, the strengths outweigh these weaknesses. Consequently, I am leaning towards suggesting acceptance of the paper. I am looking forward to the rebuttal by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the in-depth reading of our paper and for the helpful comments and suggestions.\n\n**Limitation to entities mentioned in prompt:** Yes, this is indeed a primary limitation of self-guidance as presented. However, other forms of self-guidance could help in situations where a text description is not available. For example, to move objects, one can select a point on the canvas to extract the feature vector at that location and guide that feature to appear at another location instead; one could also extract a soft segmentation map by finding feature vectors (which have been shown in concurrent work to be meaningful, e.g. [1,2]) that are epsilon-close to the “clicked” feature, rather than by going through the cross-attention map, and using those as self-guidance targets (e.g. to resize objects or change their appearance). We leave fuller exploration of these ideas to future work, though preliminary results indicate the direction is promising.\n\n**Sensitivity to hyperparameters:** We agree that more discussion on sensitivity to hyper-parameters is valuable. As also discussed in responses to reviewers EBx5 and jH2E, we are releasing an open-source implementation of the model so that the community can explore these aspects, but we also find that self-guidance is not highly sensitive to choices of weights, etc. and that once good settings for hyperparameters are found, they can remain more-or-less fixed across images. We also plan on adding a figure demonstrating the effect of varying these axes to the final manuscript, and included a figure in the global response to the reviews visualizing this.\n\n**Self-guidance terms:** We agree that further exploration of self-guidance and a more theoretically grounded approach is an exciting future direction. Though the terms we propose are indeed “heuristic”, they proved effective for the desired edits on the models we explored. The mechanistic role of attention interactions in large models remains an open research question and we see self-guidance as another tool that can expose unexpected behaviors in these models, hopefully leading to a better understanding of their internals. For example, in Figure 9 (c,d), we find hints that – somewhat counterintuitively – spatial attention patterns themselves may somehow be used to encode desired appearance information of objects.\n\n**Pixel-space vs latent-space models:** We do find that self-guidance generalizes to Stable Diffusion and are releasing an open-source implementation of such, please see global comment for more detail.\n\n**Self-guidance across architectures and modalities:** The core idea behind self-guidance is that powerful generative models learn interesting internal representations, and we can use these representations to guide sampling. This does not rely on attention, but does rely on a method for identifying useful features from internals and a way to guide sampling. We are hopeful that similar techniques can be useful for other modalities.\n\n**Appearance term:** This is a fair point. Empirically, we found that guiding other features in the UNet did not control appearance (at least as perceived by humans) as much as the features toward the very end of the model, nearest to the output head. Attention maps at that stage are also higher-resolution with sharper edges. This motivated the formulation of a single appearance term rather than a bundle throughout the depth of the network. It is possible that we could effectively control other attributes by only guiding one attention layer as well, and this may allow a modest speedup at inference time.\n\n**Other sampling methods:** We found self-guidance to work somewhat worse with DDIM. We hypothesize this is due to the higher stochasticity of DDPM being less sensitive to going off-manifold or perhaps due to the inaccuracy intrinsic to the reverse diffusion sampler [3]. That being said, in the Stable Diffusion implementation, we are able to use other schedulers such as the Euler discrete sampler [4] with success. Recent work has shown the ability to reconstruct images by inverting multiple layers of DDPM noise [5], and this may indeed be a promising avenue to further improve quality of real image editing.\n\n**Minor nitpicks:** Thanks for the attentive reading of the manuscript and we will incorporate these changes into its final edition, including citing eDiff-I and defining indices and variables in equations.\n\n* [1] Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence, Luo et al., 2023\n* [2] Emergent Correspondence from Image Diffusion, Tang et al., 2023\n* [3] Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC, ICML 2023\n* [4] Elucidating the Design Space of Diffusion-Based Generative Models, Karras et al., NeurIPS 2022\n* [5] An Edit Friendly DDPM Noise Space: Inversion and Manipulations, Huberman-Spiegelglas et al., 2023\n"}]

Author rebuttal:
We thank all reviewers for their time and detailed reading of our paper. Reviewers found our approach to controllable image generation “conceptually simple, easy to implement, computationally efficient and highly flexible” with “clear and concise presentation” in an “interesting and important research topic” with “many relevant applications”. Reviewers appreciated that, by guiding properties of the attention and features of large generative models, our method “does not require retraining while still achieving impressive results” and provides a level of control that “no previous image editing methods with text-to-image diffusion models reach”.

A shared concern was the sensitivity of our method to hyperparameters, namely the guidance scale that effectively weights each self-guidance term differently. We include a figure (Fig. 1) in the PDF attached to this response to demonstrate the robustness of self-guidance to different choices of scale hyperparameters. Note that self-guidance performs reasonably for a wide range of values, but extremely small or large scales induce artifacts or fail to execute the edit. Some reviewers were curious whether our approach generalizes to latent-space text-image diffusion models such as Stable Diffusion. We plan on releasing an open-source implementation of self-guidance on Stable Diffusion XL. We are still working on this implementation, but have included preliminary results in Fig. 2 of the attached PDF, validating that our method indeed generalizes to other diffusion architectures. We are also sharing an anonymized Colab notebook containing the current state of the code used to generate this figure, and are working hard to implement the remaining pieces.


Meta-review: This paper proposes a method for editing images using text-to-image generative diffusion models. The proposed method guides the sampling process and can guide the object's location, size, shape or appearance. The reviews highlighted the proposed methods simiplicity, novelty and practicality and strong performances. While some points about hyperparameters and results on stable diffusion were raised by the reviewers initially, they were addressed to satisfaction of reviewers in the rebuttal.
Decision: Accept (poster)