SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dave Epstein<sup>1,2</sup> Allan Jabri<sup>1</sup> Ben Poole<sup>2</sup> Alexei A. Efros<sup>1</sup> Aleksander Holynski<sup>1,2</sup>  
<sup>1</sup>UC Berkeley <sup>2</sup>Google Research
Large-scale generative models are capable of producing high-quality images from detailed text descriptions. However, many aspects of an image are difficult or impossible to convey through text. We introduce self-guidance, a method that provides greater control over generated images by guiding the internal representations of diffusion models. We demonstrate that properties such as the shape, location, and appearance of objects can be extracted from these representations and used to steer the sampling process. Self-guidance operates similarly to standard classifier guidance, but uses signals present in the pretrained model itself, requiring no additional models or training. We show how a simple set of properties can be composed to perform challenging image manipulations, such as modifying the position or size of specific objects, merging the appearance of objects in one image with the layout of another, composing objects from multiple images into one, and more. We also show that self-guidance can be used for editing real images. See our project page for results and an interactive demo: https://dave.ml/selfguidance
Generative image models have improved rapidly in recent years with the adoption of large text-image datasets and scalable architectures [7, 10, 12, 27, 28, 32, 34, 38]. These models are able to create realistic images given a text prompt describing just about anything. However, despite the incredible abilities of these systems, discovering the right prompt to generate the exact image a user has in mind can be surprisingly challenging. A key issue is that all desired aspects of an image must be communicated through text, even those that are difficult or even impossible to convey precisely.
To address this limitation, previous work has introduced methods [9, 14, 18, 30] that tune pretrained models to better control details that a user has in mind. These details are often supplied in the form of reference images along with a new textual prompt [2, 4] or other forms of conditioning [1, 31, 39]. However, these approaches all either rely on fine-tuning with expensive paired data (thus limiting the scope of possible edits) or must undergo a costly optimization process to perform the few manipulations they are designed for. While some methods [11, 21, 22, 36] can perform zero-shot editing of an input image using a target caption describing the output, these methods only allow for limited control, often restricted to structure-preserving appearance manipulation or uncontrolled image-to-image translation.
By consequence, many simple edits still remain out of reach. For example, how can we move or resize one object in a scene without changing anything else? How can we take the appearance of an object in one image and copy it over to another, or combine the layout of one scene with the appearance of a second one? How can we generate images with certain items having precise shapes at specific positions on the canvas? This degree of control has been explored in the past in smaller scale settings [5, 8, 19, 24, 37, 40], but has not been convincingly demonstrated with modern large-scale diffusion models [26, 32, 38].
(a) Original
(b) Swap objects
(c) Enlarge macaron
(d) Replace macaron
(e) Copy scene appearance
(f) Copy scene layout
(a) Original
Figure 1: Self-guidance is a method for controllable image generation that guides sampling using the attention and activations of a pretrained diffusion model. With self-guidance, we can move or resize objects, or even replace them with items from real images, without changing the rest of the scene (b-d). We can also borrow the appearance of other images or rearrange scenes into new layouts (e-f).
"a DSLR photo of a meatball and a donut falling from the clouds onto a neighborhood"
(b) Move donut
(c) Shrink donut
(d) Replace donut
We propose self-guidance, a zero-shot approach which allows for direct control of the shape, position, and appearance of objects in generated images. Self-guidance leverages the rich representations learned by pretrained text-to-image diffusion models – namely, intermediate activations and attention – to steer attributes of entities and interactions between them. These constraints can be user-specified or transferred from other images, and rely only on knowledge internal to the diffusion model. Through a variety of challenging image manipulations, we demonstrate that self-guidance using only a few simple properties allows for granular, disentangled manipulation of the contents of generated images (Figure 1). Further, we show that self-guidance can also be used to reconstruct and edit real images.
- We introduce self-guidance, which takes advantage of the internal representations of pretrained text-to-image diffusion models to provide disentangled, zero-shot control over the generative process without requiring auxiliary models or supervision.
- We find that properties such as the size, location, shape, and appearance of objects can be extracted from these representations and used to meaningfully guide sampling in a zero-shot manner.
- We demonstrate that this small set of properties, when composed, allows for a wide variety of surprisingly complex image manipulations, including control of relationships between objects and the way modifiers bind to them.
- Finally, by reconstructing captioned images using their layout and appearance as computed by self-guidance, we show that we can extend our method to editing real images.
2.1 Diffusion generative models
Diffusion models learn to transform random noise into high-resolution images through a sequential sampling process [12, 33, 35]. This sampling process aims to reverse a fixed time-dependent destructive process that corrupts data by adding noise. The learned component of a diffusion model is a neural network  $\epsilon_{\theta}$  that tries to estimate the denoised image, or equivalently the noise  $\epsilon_{t}$  that was added to create the noisy image  $z_{t} = \alpha_{t}x + \sigma_{t}\epsilon_{t}$ . This network is trained with loss:
$$
L (\theta) = \mathbb {E} _ {t \sim \mathcal {U} (1, T), \epsilon_ {t} \sim \mathcal {N} (0, \mathbf {I})} \left[ w (t) | | \epsilon_ {t} - \epsilon_ {\theta} \left(z _ {t}; t, y\right) | | ^ {2} \right], \tag {1}
$$
"ap o a giant macaron and a croissant splashing in the Seine with the Eiffel Tower in the background
where  $y$  is an additional conditioning signal like text, and  $w(t)$  is a function weighing the contributions of denoising tasks to the training objective, commonly set to 1 [12, 15]. A common choice for  $\epsilon_{\theta}$  is a U-Net architecture with self- and cross-attention at multiple resolutions to attend to conditioning text in  $y$  [28, 29, 32]. Diffusion models are score-based models, where  $\epsilon_{\theta}$  can be seen as an estimate of the score function for the noisy marginal distributions:  $\epsilon_{\theta}(z_t) \approx -\sigma_t \nabla_{z_t} \log p(z_t)$  [35].
Given a trained model, we can generate samples given conditioning  $y$  by starting from noise  $z_{T} \sim \mathcal{N}(0, I)$ , and then alternating between estimating the noise component and updating the noisy image:
$$
\hat {\epsilon} _ {t} = \epsilon_ {\theta} \left(z _ {t}; t, y\right), \quad z _ {t - 1} = \operatorname {u p d a t e} \left(z _ {t}, \hat {\epsilon} _ {t}, t, t - 1, \epsilon_ {t - 1}\right), \tag {2}
$$
where the update could be based on DDPM [12], DDIM [34], or another sampling method (see Appendix for details). Unfortunately, naively sampling from conditional diffusion models does not produce high-quality images that correspond well to the conditioning  $y$ . Instead, additional techniques are utilized to modify the sampling process by altering the update direction  $\hat{\epsilon}_t$ .
2.2 Guidance
A key capability of diffusion models is the ability to adapt outputs after training by guiding the sampling process. From the score-based perspective, we can think of guidance as composing score functions to sample from richer distributions or to introduce conditioning on auxiliary information [7, 17, 35]. In practice, using guidance involves altering the update direction  $\hat{\epsilon}_t$  at each iteration.
Classifier guidance can generate conditional samples from an unconditional model by combining the unconditional score function for  $p(z_{t})$  with a classifier  $p(y|z_{t})$  to generate samples from  $p(z_{t}|y) \propto p(y|z_{t})p(z_{t})$  [7, 35]. To use classifier guidance, one needs access to a labeled dataset and has to learn a noise-dependent classifier  $p(y|z_{t})$  that can be differentiated with respect to the noisy image  $z_{t}$ . While sampling, we can incorporate classifier guidance by modifying  $\hat{\epsilon}_{t}$ :
$$
\hat {\epsilon} _ {t} = \epsilon_ {\theta} (z _ {t}; t, y) - s \sigma_ {t} \nabla_ {z _ {t}} \log p (y | z _ {t}), \tag {3}
$$
where  $s$  is an additional parameter controlling the guidance strength. Classifier guidance moves the sampling process towards images that are more likely according to the classifier [7], achieving a similar effect to truncation in GANs [3], and can also be applied with pretrained classifiers by first denoising the intermediate noisy image (though this requires additional approximations [1]).
In general, we can use any energy function  $g(z_{t}; t, y)$  to guide the diffusion sampling process, not just the probabilities from a classifier.  $g$  could be the approximate energy from another model [17], a similarity score from a CLIP model [23], an arbitrary time-independent energy as in universal guidance [1], bounding box penalties on attention [6], or any attributes of the noisy images. We can incorporate this additional guidance alongside classifier-free guidance [13] to obtain high-quality text-to-image samples that also have low energy according to  $g$ :
$$
\hat {\epsilon} _ {t} = (1 + s) \epsilon_ {\theta} \left(z _ {t}; t, y\right) - s \epsilon_ {\theta} \left(z _ {t}; t, \emptyset\right) + v \sigma_ {t} \nabla_ {z _ {t}} g \left(z _ {t}; t, y\right), \tag {4}
$$
where  $s$  is the classifier-free guidance strength and  $v$  is an additional guidance weight for  $g$ . As with classifier guidance, we scale by  $\sigma_t$  to convert the score function to a prediction of  $\epsilon_t$ . The main contribution of our work is to identify energy functions  $g$  useful for controlling properties of objects and interactions between them.
2.3 Where can we find signal for controlling diffusion?
While guidance is a flexible way of controlling the sampling process, energy functions typically used [1, 39] require auxiliary models (adapted to be noise-dependent) as well as data annotated with properties we would like to control. Can we circumvent these costs? Recent work [11, 36] has shown that the intermediate outputs of the diffusion U-Net encode valuable information [16, 25] about the structure and content of the generated images. In particular, the self and cross-attention maps  $\{\mathcal{A}_{i,t} \in \mathbb{R}^{H_i \times W_i \times K}\}$  often encode structural information [11] about object position and shape, while the network activations  $\{\Psi_{i,t} \in \mathbb{R}^{H_i \times W_i \times D_i}\}$  allow for maintaining coarse appearance [36] when extracted from appropriate layers. While these editing approaches typically share attention and activations naively between subsequent sampling passes, drastically limiting the scope of possible manipulations, we ask: what if we tried to harness model internals in a more nuanced way?
Figure 2: Overview: We leverage representations learned by text-image diffusion models to steer generation with self-guidance. By constraining intermediate activations  $\Psi_t$  and attention interactions  $\mathcal{A}_t$ , self-guidance can control properties of entities named in the prompt. For example, we can change the position and shape of the burger, or copy the appearance of ice cream from a source image.
3 Self-guidance
Inspired by the rich representations learned by diffusion models, we propose self-guidance, which places constraints on intermediate activations and attention maps to steer the sampling process and control entities named in text prompts (see Fig. 2). These constraints can be user-specified or copied from existing images, and rely only on knowledge internal to the diffusion model.
We identify a number of properties useful for meaningfully controlling generated images, derived from the set of softmax-normalized attention matrices  $\{\mathcal{A}_{i,t} \in \mathbb{R}^{H_i \times W_i \times K}\}$  and activations  $\{\Psi_{i,t} \in \mathbb{R}^{H_i \times W_i \times D_i}\}$  extracted from the standard denoising forward pass  $\epsilon_{\theta}(z_t; t, y)$ . To control an object mentioned in the text conditioning  $y$  at token indices  $k$ , we can manipulate the corresponding attention channel(s)  $\mathcal{A}_{i,t,\dots,k} \in \mathbb{R}^{H_i \times W_i \times |k|}$  and activations  $\Psi_{i,t}$  (extracted at timestep  $t$  from a noisy image  $z_t$  given text conditioning  $y$ ) by adding guidance terms to Eqn. 4.
Object position. To represent the position of an object (omitting attention layer index and timestep for conciseness), we find the center of mass of each relevant attention channel:
$$
\operatorname {c e n t r o i d} (k) = \frac {1}{\sum_ {h , w} \mathcal {A} _ {h , w , k}} \left[ \begin{array}{l} \sum_ {h, w} w \cdot \mathcal {A} _ {h, w, k} \\ \sum_ {h, w} h \cdot \mathcal {A} _ {h, w, k} \end{array} \right] \tag {5}
$$
We can use this property to guide an object to an absolute target position on the image. For example, to move "burger" to position  $(0.3, 0.5)$ , we can minimize  $\|(0.3, 0.5) - \text{centroid}(k)\|_1$ . We can also perform a relative transformation, e.g., move "burger" to the right by  $(0.1, 0.0)$  by minimizing  $\|\text{centroid}_{\text{orig}}(k) + (0.1, 0.0) - \text{centroid}(k)\|_1$ .
Object size. To compute an object's size, we spatially sum its corresponding attention channel:
$$
\operatorname {s i z e} (k) = \frac {1}{H W} \sum_ {h, w} \mathcal {A} _ {h, w, k} \tag {6}
$$
In practice, we find it beneficial to differentiably threshold the attention map  $\mathcal{A}_{\mathrm{thresh}}$  before computing its size, to eliminate the effect of background noise. We do this by taking a soft threshold at the midpoint of the per-channel minimum and maximum values (see Appendix for details). As with position, one can guide to an absolute size (e.g. half the canvas) or a relative one (e.g.  $10\%$  larger).
Object shape. For even more granular control than position and size, we can represent the object's exact shape directly through the thresholded attention map itself:
$$
\operatorname {s h a p e} (k) = \mathcal {A} _ {k} ^ {\text {t h r e s h}} \tag {7}
$$
This shape can then be guided to match a specified binary mask (either provided by a user or extracted from the attention from another image) with  $\| \text{target\_shape} - \text{shape}(k)\|_1$ . Note that we can
"a photo of a fluffy cat sitting on a museum bench looking at an oil painting of cheese"
(a) Original
Figure 3: Moving and resizing objects. By only changing the properties of one object (as in Eqn. 9), we can move or resize that object without modifying the rest of the image. In these examples, we modify "massive sun", "oil painting of cheese", and "raccoon in a barrel", respectively. apply any arbitrary transformation (scale, rotation, translation) to this shape before using it as a guidance target, which allows us to manipulate objects while maintaining their silhouette.
"a photo of a raccoon in a barrel going down a waterfall"
(b) Move up
(c) Move down
(d) Move left
Fix all other object shapes
(e) Move right
(f) Shrink
(g) Enlarge
Object appearance. Considering thresholded attention a rough proxy for object extent, and spatial activation maps as representing local appearance (since they ultimately must be decoded into an unnoised RGB image), we can reach a notion of object-level appearance by combining the two:
$$
\operatorname {a p p e a r a n c e} (k) = \frac {\sum_ {h , w} \operatorname {s h a p e} (k) \odot \Psi}{\sum_ {h , w} \operatorname {s h a p e} (k)} \tag {8}
$$
4 Composing self-guidance properties
The small set of properties introduced in Section 3 can be composed to perform a wide range of image manipulations, including many that are intractable through text. We showcase this collection of manipulations and, when possible, compare to prior work that accomplishes similar effects. All experiments were performed onImagen [32], producing  $1024 \times 1024$  samples. For more samples and details on the implementation of self-guidance, please see the Appendix.
Adjusting specific properties. By guiding one property to change and all others to keep their original values, we can modify single objects in isolation (Fig. 3b-3e). For a caption  $C = y$  with words at indices  $\{c_i\}$ , in which  $O = \{o_j\} \subseteq C$  are objects, we can move an object  $o_k$  at time  $t$  with:
$$
\begin{array}{l} g = w _ {0} \frac {1}{| O | - 1} \sum_ {o \neq o _ {k} \in O} \frac {1}{| \mathcal {A} |} \sum_ {i = 0} ^ {| \mathcal {A} |} \| \mathrm {s h a p e} _ {i, t, \mathrm {o r i g}} (o) - \mathrm {s h a p e} _ {i, t} (o) \| _ {1} \\ + w _ {1} \overbrace {\frac {1}{| O |} \sum_ {o \in O} \| \text {a p p e a r a n c e} _ {t , \text {o r i g}} (o) - \text {a p p e a r a n c e} _ {t} (o) \| _ {1}} ^ {\text {F i x a l l a p p e a r a n c e s}} \tag {9} \\ + w _ {2} \overbrace {\frac {1}{| \mathcal {A} |} \sum_ {i = 0} ^ {| \mathcal {A} |} \| \mathcal {T} \left(\operatorname {s h a p e} _ {i , t , \text {o r i g}} \left(o _ {k}\right)\right) - \operatorname {s h a p e} _ {i , t} \left(o _ {k}\right) \| _ {1}} ^ {\text {G u i d e o} _ {k} \text {’ s s h a p e t o t r a n s l a t e d o r i g i n a l s h a p e}} \\ \end{array}
$$
Where  $\mathsf{shape}_{\mathrm{orig}}$  and shape are extracted from the generation of the initial and edited image, respectively. Critically,  $\mathcal{T}$  lets us define whatever transformation of the  $H_{i} \times W_{i}$  spatial attention map we
"distant shot of the tokyo tower with a massive sun in the sky"
(b) New appearances
Figure 4: Sampling new appearances. By guiding object shapes (Eqn. 7) towards reconstruction of a given image's layout (a), we can sample new appearances for a given scene (b-d).
(c) ControlNet [39]
(d) PtP [11]
want. To move an object,  $\mathcal{T}$  translates the attention mask the desired amount. We can also resize objects (Fig. 3f-3g) with Eqn. 9 by changing  $\mathcal{T}$  to up- or down-sample shape matrices.
Constraining per-object layout but not appearance finds new "styles" for the same scene (Fig. 4):
$$
g = w _ {0} \overbrace {\frac {1}{| O |} \sum_ {o \in O} \frac {1}{| A |} \sum_ {i = 0} ^ {| A |} \| \mathrm {s h a p e} _ {i , t , \text {o r i g}} (o) - \mathrm {s h a p e} _ {i , t} (o) \| _ {1}} ^ {\text {F i x a l l o b j e c t s h a p e s}} \tag {10}
$$
We can alternatively choose to guide all words, not just nouns or objects, changing summands to  $\sum_{c\neq o_k\in C}$  instead of  $\sum_{c\neq o_k\in O}$ . See Appendix for further discussion.
Composition between images. We can compose properties across multiple images into a cohesive sample, e.g. the layout of an image  $A$  with the appearance of objects in another image  $B$  (Fig. 5):
$$
g = w _ {0} \overbrace {\frac {1}{| O |} \sum_ {o \in O} \frac {1}{| A |} \sum_ {i = 0} ^ {| A |} \| \mathrm {s h a p e} _ {i , t , A} (o) - \mathrm {s h a p e} _ {i , t} (o) \| _ {1}} ^ {\text {C o p y o b j e c t s h a p e s f r o m A}} \tag {11}
$$
$$
+ w _ {1} \overbrace {\frac {1}{| O |} \sum_ {o \in O} \| \text {a p p e a r a n c e} _ {t , B} (o) - \text {a p p e a r a n c e} _ {t} (o) \| _ {1}} ^ {\text {C o p y o b j e c t a p p e a r a n c e f r o m B}}
$$
We can also borrow only appearances, dropping the first term to sample new arrangements for the same objects, as in the last two columns of Figure 5.
Highlighting the compositionality of self-guidance terms, we can further inherit the appearance and/or shape of objects from several images and combine them into one (Fig. 6). Say we have  $J$  images, where we are interested in keeping a single object  $o_{k_j}$  from each one. We can collage these objects "in-place" - i.e. maintaining their shape, size, position, and appearance - straightforwardly:
$$
g = w _ {0} \overbrace {\frac {1}{J} \sum_ {j} \frac {1}{| \mathcal {A} |} \sum_ {i = 0} ^ {| \mathcal {A} |} \| \mathrm {s h a p e} _ {i , t , j} \left(o _ {k _ {j}}\right) - \mathrm {s h a p e} _ {i , t} \left(o _ {k}\right) \| _ {1}} ^ {\text {C o r p y e a c h o b j e c t ' s s h a p e , p o s i t i o n , a n d s i z e}} \tag {12}
$$
$$
+ w _ {1} \overbrace {\frac {1}{J} \sum_ {j} \| \text {a p p e a r a n c e} _ {t , j} \left(o _ {k _ {j}}\right) - \text {a p p e a r a n c e} _ {t} \left(o _ {k}\right) \| _ {1}} ^ {\text {C o p y e a c h o b j e c t s a p p e a r a n c e}}
$$
We can also take only the appearances of the objects from these images and copy the layout from another image, useful if object positions in the  $J$  images are not mutually compatible (Fig. 6f).
"a photo of a parrot riding a horse down a city street"
Figure 5: Mix-and-match. By guiding samples to take object shapes from one image and appearance from another (Eqn. 11), we can rearrange images into layouts from other scenes. Input images are along the diagonal. We can also sample new layouts of a scene by only guiding appearance (right).
"a photo of a suitcase, a bowling ball, and a phone washed up on a beach after a shipwreck"
← Layout →
No layout conditioning
"a photo of a picnic blanket, a fruit tree, and a car by the lake"
(a) Take blanket
(b) Take tree
(c) Take car
(d) Result
(f) Final result
(a) Take matcha
(b) Take kettle
(c) Take fruit
(e) + Target layout
(d) Result
(e) + Target layout
"a photo of a dog wearing a knit sweater and a baseball cap drinking a cocktail"
(a) Take sweater
(b) Take cocktail
(c) Take cap
(d) Result*
Figure 6: Compositional generation. A new scene (d) can be created by collaging objects from multiple images (Eqn. 12). Alternatively - e.g. if objects cannot be combined at their original locations due to incompatibilities in these images' layouts (*as in the bottom row) - we can borrow only their appearance, and specify layout with a new image (e) to produce a composition (f) (Eqn. 19).
Figure 7: Appearance transfer from real images. By guiding the appearance of a generated object to match that of one in a real image (outlined) as in Eqn. 13, we can create scenes depicting an object from real life, similar to DreamBooth [30], but without any fine-tuning and only using one image.
Figure 8: Real image editing. Our method enables the spatial manipulation of objects (shown in Figure 3 for generated images) for real images as well.
Editing with real images. Our approach is not limited to only images generated by a model, whose internals we have access to by definition. By running  $T$  noised versions of a (captioned) existing image through a denoiser - one for each forward process timestep - we extract a set of intermediates that can be treated as if it came from a reverse sampling process (see Appendix for more details). In Fig. 8, we show that, by guiding shape and appearance for all tokens, we generate faithful reconstructions of real images. More importantly, we can manipulate these real images just as we can generated ones, successfully controlling properties such as appearance, position, or size. We can also transfer the appearance of an object of interest into new contexts (Fig. 7), from only one source image, and without any fine-tuning:
$$
g = w _ {0} \overbrace {\left\| \text {a p p e a r a n c e} _ {t , \text {o r i g}} \left(o _ {k _ {\text {o r i g}}}\right) - \text {a p p e a r a n c e} _ {t} \left(o _ {k}\right) \right\| _ {1}} ^ {\text {C o p y o b j e c t a p p e a r a n c e}} \tag {13}
$$