SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Attributes and interactions. So far we have focused only on the manipulation of objects, but we can apply our method to any concept in the image, as long as it appears in the caption. We demonstrate manipulation of verbs and adjectives in Fig. 9, and show an example where certain self-guidance constraints can help in enforcing attribute binding in the generation process.
We introduce a method for guiding the diffusion sampling process to satisfy properties derived from the attention maps and activations within the denoising model itself. While we propose a number of
"a cat and a monkey laughing on a road"
Move laughing to the right
(a) Original
ng to the right
(b) Modified
Change messy location
(c) At  $\langle 0.3,0.6\rangle$
(d) At  $\langle 0.8, 0.8 \rangle$
"green hat, blue book, yellow shoes, red jacket"
Move red to jacket, yellow to shoes
(e) Original
(f) Fixed
Figure 9: Manipulating non-objects. The properties of any word in the input prompt can be manipulated, not only nouns. Here, we show examples of relocating adjectives and verbs. The last example shows a case in which additional self-guidance can correct improper attribute binding.
"a photo of a squirrel trying to catch a lime mid-air"
Appearance features leak layout
(a) Unguided
(b) "lime" guided
Figure 10: Limitations. Setting high guidance weights for appearance terms tends to introduce unwanted leakage of object position (a-b). Similarly, while heavily guiding the shape of one word simply matches that object's layout as expected, high guidance on the shapes of all tokens results in a leak of appearance information (c-d). Finally, in some cases, objects are entangled in attention space, making it difficult to control them independently (e-f).
"a picture of a cake"
Multi-token layout leaks appearance
(c) Real image
(d) Layout guided
"a potato sitting on a couch with a bowl of popcorn watching football on TV"
Interacting objects entangled
(e) Original
(f) Move potato  $\rightarrow$
such properties, many more certainly exist, as do alternative formulations of those presented in this paper. Among the proposed collection of properties, a few limitations stand out.
The reliance on cross-attention maps imposes restrictions by construction, precluding control over any object that is not described in the conditioning text prompt and hindering fully disentangled control between interacting objects due to correlations in attention maps (Fig. 10e-10f). Selectively applying attention guidance at certain layers or timesteps may result in more effective disentanglement.
Our experiments also show that appearance features often contain undesirable information about spatial layout (Fig. 10a-10b), perhaps since the model has access to positional information in its architecture. The reverse is also sometimes true: guiding the shape of multiple tokens occasionally betrays the appearance of an object (Fig. 10c-10d), implying that hidden high-frequency patterns arising from interaction between attention channels may be used to encode appearance. These findings suggest that our method could serve as a window into the inner workings of diffusion models and provide valuable experimental evidence to inform future research.
The use-cases showcased in this paper, while transformative for creative uses, carry the risk of producing harmful content that can negatively impact society. In particular, self-guidance allows for a level of control over the generation process that might enable potentially harmful image manipulations, such as pulling in the appearance or layout from real images into arbitrary generated content (e.g., as in Fig. 7). One such dangerous manipulation might be the injection of a public figure into an image containing illicit activity. In our experiments, we mitigate this risk by deliberately refraining from generating images containing humans. Additional safeguards against these risks include methods for embedded watermarking [20] and automated systems for safe filtering of generated imagery.
We thank Oliver Wang, Jason Baldridge, Lucy Chai, and Minyoung Huh for their helpful comments. Dave is supported by the PD Soros Fellowship. Dave and Allan conducted part of this research at Google, with additional funding provided by DARPA MCS and ONR MURI.
[1] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. arXiv preprint arXiv:2302.07121, 2023.
[2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 707-723. Springer, 2022.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1809.11096, 2018.
[4] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
[5] Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in gans. arXiv preprint arXiv:2103.10426, 2021.
[6] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373, 2023.
[7] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.05233, 2021.
[8] Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. Blobgan: Spatially disentangled scene representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 616-635. Springer, 2022.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.
[10] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv:2206.09012, 2022.
[11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022.
[14] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagc: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276, 2022.
[15] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurIPS, 2021.
[16] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960, 2022.
[17] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, 2022.
[18] Vivian Liu and Lydia B Chilton. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-23, 2022.
[36] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572, 2022.
[37] Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905, 2021.
[38] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv:2206.10789, 2022.
[39] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[40] Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, and Qifeng Chen. Region-based semantic factorization in gans. arXiv preprint arXiv:2202.09649, 2022.
A. Implementation details
We apply our self-guidance term following best practices for classifier-free guidance on Imagen [32]. Specifically, where  $N$  is the number of DDPM steps, we take the first  $\frac{3N}{16}$  steps with self-guidance and the last  $\frac{N}{32}$  without. The remaining  $\frac{25N}{32}$  steps are alternated between using self-guidance and not using it. We use  $N = 1024$  steps. Our method works with 256 and 512 steps as well, though self-guidance weights occasionally require adjustment. We set  $v = 7500$  in Eqn. 4 as an overall scale for gradients of the functions  $g$  defined below — we find that the magnitude of per-pixel gradients is quite small (often in the range of  $10^{-7}$  to  $10^{-6}$ , so such a high weight is needed to induce changes.
We apply centroid, size, and shape terms on all cross-attention interactions in the model we use. In total, there are 36 of these, across the encoder, bottleneck, and decoder, at  $8 \times 8$ ,  $16 \times 16$ , and  $32 \times 32$  resolutions. We apply the appearance term using the activations of the penultimate layer in the decoder (two layers before the prediction readout) and the final cross-attention operation. We experimented with features from other parts of the U-Net denoiser, namely early in the encoder before positional information can propagate through the image (to prevent appearance layout entanglement), but found these to work significantly worse. To avoid degenerate solutions, we apply a stop-gradient to the attention in the appearance term so only information about activations is back-propagated. We take the mean spatially of all shape terms and across activation dimensions for all appearance terms, which we omit in all equations for conciseness.
Attention mask binarization. In practice, it is beneficial to differentiably binarize the attention map (with sharpness controlled by  $s$ ) before computing its size or utilizing its shape, to eliminate the effect of background noise (this is empirically less important when guiding centroids, so we do not binarize in that case). We do this by taking a soft threshold at the midpoint of the per-channel minimum and maximum values. More specifically, we apply a shifted sigmoid on the attention normalized to have minimum 0 and maximum 1, followed by another such normalization to ensure the high value is 1 and the low 0 after applying the sigmoid. We use  $s = 10$  and redefine Eqn. 6.
$$
\operatorname {n o r m a l i z e} (\mathbf {X}) = \frac {\mathbf {X} - \min  _ {h , w} (\mathbf {X})}{\max  _ {h , w} (\mathbf {X}) - \min  _ {h , w} (\mathbf {X})} \tag {14}
$$
$$
\mathcal {A} ^ {\text {t h r e s h}} = \text {n o r m a l i z e} (\text {s i g m o i d} (s \cdot (\text {n o r m a l i z e} (\mathcal {A}) - 0. 5))) \tag {15}
$$
$$
\operatorname {s i z e} (k) = \frac {1}{H W} \sum_ {h, w} \mathcal {A} _ {h, w, k} ^ {\text {t h r e s h}} \tag {16}
$$
"a photo of a carrot and an onion in a hot tub outdoors"
"a photo of an oak tree and a pineapple outside an arctic igloo"
Figure 11: Moving objects. Non-cherry-picked results for moving objects in scenes using Eqn. 9. We move onion down and to the right, pineapple to the right, and owl up and pig down, respectively. All scenes use weights  $w_{0} = 1.5$ ,  $w_{1} = 0.25$ , and  $w_{2} = 2$ .
(b) Edited
"a photo of a chicken walking across the street with an Italian sports car waiting for it"
(a) Original
Figure 12: Resizing objects. Non-cherry-picked results for resizing objects in scenes using Eqn. 9, with  $\mathcal{T}$  specified to up- or down-sample attention maps. We reduce the punching bag's height  $0.5\times$  and enlarge chicken  $2.5\times$  and boombox  $2\times$ . All scenes use  $w_{0} = 2$ ,  $w_{1} = 0.25$ , and  $w_{2} = 3$ .
Fix all other object shapes
Fix all object appearances
Fix  $o_k$  's size
B. Using self-guidance
Maximizing consistency. In general, we find that sharing the same sequence of noise in the DDPM process between an image and its edited version is not necessary to maintain high levels of consistency, but can help if extreme precision is desired. We find that maintaining object silhouettes under transformations such as resizing and repositioning is more effective if applying a transformation  $\mathcal{T}$  to the original shape, rather than expressing the same change through centroid and size.
Guiding "background" words. To keep all objects of the scene fixed but one (Fig. 3), one can either guide all other tokens in the prompt (including "a photo of" and other abstract terms) to keep their shape, or only select the other salient objects and hold those fixed. In general, since abstract words are often used for message passing and have attention patterns that are correlated with the layout of the scene, we prefer not to guide their layouts to maximize compositionality.
Mitigating appearance-Layout entanglement. When words or concepts span multiple tokens, we can mean-pooling attention maps across these tokens before processing them, though do not find this to improve results. We also find that corrupting target shapes with Gaussian noise helps mitigate this effect, providing some evidence for this hypothesis.
Moving objects. We use  $w_0 \in [0.5, 2]$ ,  $w_1 \in [0.03, 0.3]$ ,  $w_2 \in [0.5, 5]$  in Eqn. 9. Alternatively, we can express  $o_k$ 's new location through its centroid, adding a term to keep size fixed:
$$
g = w _ {0} \frac {1}{| O | - 1} \sum_ {o \neq o _ {k}} \frac {1}{| A |} \sum_ {i = 0} ^ {| A |} \| \mathsf {s h a p e} _ {i, t, \mathrm {o r i g}} (o) - \mathsf {s h a p e} _ {i, t} (o) \| _ {1}
$$
$$
+ w _ {1} \overbrace {\frac {1}{| O |} \sum_ {o \in O} \| \text {a p p e a r a n c e} _ {t , \text {o r i g}} (o) - \text {a p p e a r a n c e} _ {t} (o) \| _ {1}} ^ {1}
$$
$$
\begin{array}{l} + w _ {2} \frac {1}{| \mathcal {A} |} \sum_ {i = 0} ^ {| \mathcal {A} |} \| \operatorname {s i z e} _ {i, t, \text {o r i g}} (o _ {k}) - \operatorname {s i z e} _ {i, t} (o _ {k}) \| _ {1} \\ + w _ {3} \underbrace {\frac {1}{| \mathcal {A} |} \sum_ {i = 0} ^ {| \mathcal {A} |} \| \text {t a r g e t} _ {-} \text {c e n t r o i d} - \text {c e n t r o i d} _ {i , t} (o _ {k}) \| _ {1}} _ {\text {C h a n g e} o _ {k} \text {’ s p o s i t i o n}} \\ \end{array}
$$
"a photo of a kangaroo and a punching bag at the gym"
Figure 13: Creating new appearances for scenes. Non-cherry-picked results sampling different "styles" of appearances given the same layout, using Eqn. 10. We use  $w_0 = 0.7$ , 0.3, and 0.3 respectively for each result, to preserve greater structure in the background of the first picture.
Where target_centroid can be computed as a shifted version of the timestep-and-attention-specific centroid<sub>orig</sub> if desired, or selected to be an absolute value on the canvas (repeated across all timesteps). We generally use weights  $w_0 \in [0.5, 2]$ ,  $w_1 \in [0.03, 0.3]$ ,  $w_2 \in [0.5, 2]$ ,  $w_3 \in [1, 3]$ .
Resizing objects. We can follow Eqn. 9 to resize objects as well, by setting  $\mathcal{T}$  to upsample or downsample the original mask. We can similarly use Eqn. 17, omitting the final term and setting the target size to a desired value, either computed as a function of  $\mathrm{size}_{\mathrm{orig}}(o_k)$  or provided as an absolute proportion of pixels on the canvas that the object should cover. We use the same weight range for all weights except we set  $w_{2} \in [1,3]$ ,  $w_{3} = 0$  for Eqn. 17.
Sampling new appearances. We set  $w_0 \in [0.1, 1]$  in Eqn. 10. Generally, higher values lead to extremely precise layout preservation at the expense of diversity in appearance.
Sampling new layouts. Just as we can find new appearances for a scene of a given layout, we can perform the opposite operation, finding new layouts for scenes where objects have a given appearance:
$$
g = w _ {0} \overbrace {\frac {1}{| O |} \sum_ {o \in O} \| \text {a p p e a r a n c e} _ {t , \text {o r i g}} (o) - \text {a p p e a r a n c e} _ {t} (o) \| _ {1}} ^ {\text {F i x a l l a p p e r a n c e s}} \tag {18}
$$
We almost always use  $w_0 \in [0.05, 0.25]$ .
Collaging objects in-place. Eqn. 12 can be easily generalized to more than one object per image (adding another sum across all objects) or to the case where prompts vary between images (mapping from  $k_{j}$  to the corresponding indices in the new image). We set  $w_{0} \in [0.5, 1]$ ,  $w_{1} \in [0.05, 0.3]$ .
Collaging objects with a new layout. As shown in Fig. 6f, we can also collage objects into a new layout specified by a target image  $J + 1$ , in addition to the  $J$  images specifying object appearance:
$$
g = w _ {0} \overbrace {\frac {1}{| \mathcal {A} |} \sum_ {i = 0} ^ {| \mathcal {A} |} \| \operatorname {s h a p e} _ {i , t , J + 1} \left(o _ {k _ {J + 1}}\right) - \operatorname {s h a p e} _ {i , t} \left(o _ {k}\right) \| _ {1}} ^ {\text {C o p y a l l o b j e c t s h a p e s}} + w _ {1} \overbrace {\frac {1}{J} \sum_ {j} \| \operatorname {a p p e r a n c e} _ {t , j} \left(o _ {k _ {j}}\right) - \operatorname {a p p e r a n c e} _ {t} \left(o _ {k}\right) \| _ {1}} ^ {\text {C o p y e a c h o b j e c t ' s a p p e r a n c e}} \tag {19}
$$
As in Eqn. 12, we set  $w_0 \in [0.5, 1]$ ,  $w_1 \in [0.05, 0.3]$ .
Figure 14: Creating new layouts for scenes. Non-cherry-picked results sampling new layouts for the same scenes, using Eqn. 18. We use  $w_0 = 0.07$ , 0.07, and 0.2 respectively.
Figure 15: Appearance transfer from real images. Non-cherry picked results sampling new images with a given object's appearance specified by a real images, as in Eqn. 13. We use  $w_0 = 0.15$ .
Transferring object appearances to new layouts. Nothing requires the indices (or in fact, the objects those indices refer to) to be the same in the image being generated and the original image being used as a source, as long as there is a mapping specified between the indices in the old and new images which should correspond. Call this mapping  $m$ . We can then take the appearance of an object  $o_k$  in a source image and transfer it to an image with any new prompt as follows, as specified in Eqn. 13 (with typical weights  $w_0 \in [0.01, 0.1]$ ):
$$
g = w _ {0} \overbrace {\left\| \text {a p p e a r a n c e} _ {t , \text {o r i g}} \left(o _ {k}\right) - \text {a p p e a r a n c e} _ {t} \left(m \left(o _ {k}\right)\right) \right\| _ {1}} ^ {\text {C o p y o b j e c t a p p e a r a n c e}} \tag {20}
$$
Merging layout and appearance. We use  $w_0 \in [1,2]$  and  $w_1 \in [0.1,0.3]$  in Eqn. 11.
Editing with real images. Importantly, our method is not limited to editing generated images to whose internals it has access by definition. We find that we can also meaningfully guide generation
(b) Ours
"a dslr photo of a teapot floating in the sea"
Figure 16: Ablating appearance transfer from real images. To verify the efficacy of our approach, we compare our results from Fig. 7 in the paper to random samples from the same prompt without appearance transfer. We can see that appearance of objects varies significantly without self-guidance.
(b) Random samples without self-guidance
using the attention and activations extracted from a set of forward-process denoisings of a real image (given a caption) to "approximate" the reverse process, despite any mismatch in distributions one might imagine. Concretely, we generate  $T$  corrupted versions of a real image  $x$ ,  $\{\alpha_{t}x + \sigma_{t}\epsilon_{t}\}_{1}^{T}$ , where  $\epsilon_{t} \sim \mathcal{N}(0,1)$ . We then extract the attention  $\mathcal{A}_t$  and activations  $\Psi_t$  from the denoising network at each of these timesteps in parallel and concatenate them into a length-  $T$  sequence. We treat this sequence identically to a sequence of  $T$  internals given by subsequent sampling steps, and can thus transfer the appearance of objects from real images, output images that look like real images with moved or resized objects, and so on.
In Fig. 7, the prompts we use to transfer appearance are "A photo of a Chow Chow..." and "A DSLR photo of a teapot...". While our method works on less specific descriptions as well, it is not as reliable when object appearance is more out-of-distribution. For context, we show unguided samples under the prompts from Fig. 7 in Fig. 16, which still deviate significantly from the desired appearance, showing the efficacy of our approach. A weakness of our simple approach is that it has no constraints on the shape of the generated objects, which we leave to future work.
Weight selection heuristics. We find weights that work well to remain more or less consistent across different images given an edit, but ideal weights do vary somewhat (within predictable ranges) between different combinations of terms. Our heuristics for weight selection per term are: the more weights there are, the higher per-term weights can be without causing artifacts (and indeed, need to be, to provide ample contribution to the final result); appearance terms should have weights 1 or 2 orders of magnitude lower than layout terms; layout summary statistics (centroid and size) should have slightly lower weights than terms on the per-pixel shape; total weight of terms should not add up to more than  $\sim 5$  to avoid artifacts.
C. Additional results
We show further non-cherry-picked results for the edits we show in the main paper. Our general protocol consists of selecting an interesting prompt manually, verifying that our model creates compelling samples aligning with this prompt without self-guidance, beginning with the typical weights we use for an edit, and trying around 3-5 other weight configurations to find the one that works best for the prompt - in most cases, this is the starting set of weights. Then, we use the first 8 images we generate, without further filtering. We generate all results with different seeds to showcase the strength of guidance even without shared DDPM noise. We show more results for moving (Fig. 11) and resizing (Fig. 12) objects, sampling new appearances for given layouts (Fig. 13) as well as new layouts for a given set of objects (Fig. 14), and transferring the appearance of real objects into new contexts (Fig. 15). We also include an ablation on hyperparameter values (Fig. 17) as well as preliminary results of an implementation of self-guidance on an open-source diffusion model in Fig. 18.
"a photo of a chow chow wearing a superman outfit"
Figure 17: Hyperparameter sweeps. We show results for two edits (moving and resizing objects) using Eqn. 4 in the Supp. Mat., for different values of weights on the three edit terms, holding the other terms to the value in the middle column. Please zoom in to view in more detail. Reasonable values in the middle columns (within the expected range) lead to overall successful image manipulation. Very large hyperparameter values cause visual artifacts to appear (by moving sampling off-manifold) while still tending to perform the edit successfully, while extremely small values often fail to conduct the edit, inducing artifacts resulting from a "half-executed" manipulation.
Figure 18: Self-guidance on Stable Diffusion XL. To highlight the generality of our approach, we demonstrate preliminary results for controllable generation on a popular latent-space text-to-image diffusion model, using 100 DDPM steps (applying self-guidance from step 10 to step 90). We get best results only guiding attention in the second decoder block of the denoiser model.