SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[29] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[30] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[31] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
[32] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[33] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on a single image. ACM Trans. Graph., 2022.
[34] Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ACL, 2016.
[35] Qiucheng Wu, Yujuan Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
A Implementation of SEGA
The implementation of SEGA builds on the Stable Diffusion pipeline from the diffusers library. As describe in Sec. 3 we calculate an additional noise estimate for each semantic concept. Consequently, we make one pass through the models U-Net for each concept description. Our implementation is included in the supplementary material, together with the necessary code and hyper-parameters to generate examples shown in the paper.
Additionally, we also provide the pseudo-code notation of SEGA in Alg 1. Please note that this code is slightly simplified to use one single warm-up period  $\delta$  for all edit prompts  $e_i$ .
B Numerical Properties of Noise Estimates
As discussed in Sec. 3.2 the numerical values of noise estimates are generally Gaussian distributed. This can be attributed to the fact that they are trained to estimate a Gaussian sampled  $\epsilon$ . We plotted exemplary numerical distributions of three noise estimates in Fig. 7. These estimates are an unconditioned  $\epsilon_{\theta}(\mathbf{z}_t)$ , a text conditioned  $\epsilon_{\theta}(\mathbf{z}_t, \mathbf{c}_p)$ , and an edit conditioned  $\epsilon_{\theta}(\mathbf{z}_t, \mathbf{c}_e)$  one. All calculated at diffusion step 20. While these estimates lead to substantially different images, it is clear that their overall numerical distribution is almost identical.
Algorithm 1 Semantic Guidance (SEGA)
<sup>5</sup>https://github.com/huggingface/diffusers
Please note that this notation makes the simplifying assumption of one single warm-up period  $\delta$  for all edit prompts  $e_i$ .
Require: model weights  $\theta$ , text condition  $text_{p}$ , edit texts  $\mathbf{List}(text_{e})$  and diffusion steps  $T$
Ensure:  $s_m \in [0,1], \nu_{t=0} = 0, \beta_m \in [0,1), \lambda^i \in (0,1), s_e^i \in [0,20], s_g \in [0,20], \delta \in [0,20], t = 0$
DM  $\leftarrow$  init-diffusion-model(  $\theta$ $c_{p}\gets$  DM.encode  $(text_{p})$    
List  $(c_e)\gets$  DMencode(List  $(\text{t e x t e} _ { e } )$    
latents  $\leftarrow$  DM.sample(seed)   
while  $t\neq T$  do   
 $n_0,n_p\gets$  DM.predict-noise(latents,  $c_{p}$    
List  $(n_e)\gets$  DM.predict-noise(latents, List  $(c_e)$  ）   
 $g = s_g*(n_p - n_\emptyset)$ $i\gets 0$    
for all  $n_e^i$  in List  $(n_e)$  do   
 $\phi_t^i\gets n_e^i -n_\emptyset$  if negative guidance then  $\phi_t^i\gets -\phi_t^i$  end if   
 $\mu_t^i\gets 0$ $\mu_t^i\gets$  where  $(|\phi_t^i)|\geq \lambda_e^i,s_e^i)$ $\gamma_t^i\gets \mu_t^i\cdot \phi_t^i$ $i\gets i + 1$    
end for   
 $\gamma_t\gets \sum_{i\in I}g_i*\gamma_t^i$ $\gamma_t\gets \gamma_t + s_m*\nu_t$ $\nu_{t + 1}\gets \beta_m*\nu_t(1 - \beta_m)*\gamma_t$    
if  $t\geq \delta$  then   
 $g\gets g + \gamma_t$    
end if   
pred  $\leftarrow n_{\emptyset} + g$    
latents  $\leftarrow$  DM.update-latents(pred, latents)   
 $t\gets t + 1$    
end while
Figure 7: Distribution of numerical values in different noise estimates. All estimates follow a Gaussian distribution disregarding their different conditioning. Distribution plots calculated using kernel-density estimates with Gaussian smoothing. (Best viewed in color)
Figure 8: Uniqueness of guidance vectors. The vector for 'glasses' is calculated once on the blue-marked image and subsequently applied to other prompts (without colored border). On images where no glasses are added the guidance vector produces no visual artifacts. (Best viewed in color)
C Further intuition and ablations on hyper-parameters
Let us provide some more detailed intuition for each of SEGA's hyper-parameters
Scale  $s_e$ . The magnitude of the expression of a concept in the edited image scales monotonically with  $s_e$ . Overall, the scale behaves very robustly for a larger range of values. On the one hand, values of around 3 or 4 are sufficient for delicate changes. On the other hand,  $s_e$  is less susceptible to producing image artifacts for high values and can often be scaled to values of  $20 + \mathrm{w/o}$  quality degradation if required for the desired expression of the concept.
Threshold  $\lambda$ . SEGA automatically identifies the relevant regions of an image for each particular set of instructions and the original image in each diffusion step. Image regions outside these implicit masks (cf. Fig. 9) will not be altered by SEGA, leading to minimal changes.  $\lambda$  corresponds to the portion of the image left untouched by SEGA. Consequently, higher values of  $\lambda$  close to 1 allow for minor alterations, whereas lower values may affect larger areas of the image. Depending on the edit,  $\lambda >= 0.95$  will be sufficient for most edits, such as small changes, whereas alterations of the entire image, like style transfer, usually require values between 0.8 and 0.9.
Warmup  $\delta$ . In line with previous research, we have observed that the overall image composition is largely generated in early diffusion steps, with later ones only refining smaller details. Consequently, the number of warmup steps may be used to steer the granularity of compositional changes. Higher values of  $\delta > = 15$  will mostly change details and not composition (e.g., style transfer). Whereas strong compositional editing will require smaller values, i.e.,  $\delta > = 5$ .
Momentum. Contrary to previous hyper-parameters, momentum is more of an optional instrument to further refine the quality of the generated image. Most edits produce satisfactory results without the use of momentum, but image fidelity can further improve when used. Importantly, its main use
(a) Original image generated for prompt 'a photo of the face of a young man'
(b) Edited image using edit concept 'glasses'.
(c) Binary visualization of activating latent dimensions above SEGA threshold at increasing diffusion steps. The dimensions of the latent edited by SEGA are closely correlated with the region of the image in which the glasses are added.
case is in combination with warmup. Since momentum is already gathered during the warmup period, higher momentum facilitates higher warmup periods in combination with more radical changes.
We subsequently present visual ablations highlighting the effect of choosing different hyper parameter values. All images are generated from the same original image (shown in Fig. 10) obtained by the prompt 'a house at a lake'.
Figure 9: Visualization of latent spatial correlation of SEGA thresholding.
Figure 10: Unedited image generated from the prompt 'a house at a lake'. Included for reference of subsequent ablations.
C.1 Warmup-Scale
Visualization of the interplay between warmup  $\delta$  and semantic guidance scale  $s_e$  is shown in Fig. 11.
C.2 Warmup-Threshold
Visualization of the interplay between warmup  $\delta$  and threshold  $\lambda$  is shown in Fig. 12.
C.3 Threshold-Scale
Visualization of the interplay between threshold  $\lambda$  and semantic guidance scale  $s_e$  is shown in Fig. 13.
C.4 Momentum
Visualization of the interplay between the momentum scale  $s_m$  and momentum  $\beta$  is shown in Fig. 14.
Figure 11: Visualization of the interplay between warmup  $\delta$  and semantic guidance scale  $s_e$ .
Figure 12: Visualization of the interplay between warmup  $\delta$  and threshold  $\lambda$ .
Figure 13: Visualization of the interplay between threshold  $\lambda$  and semantic guidance scale  $s_e$ .
Figure 14: Visualization of the interplay between the momentum scale  $s_m$  and momentum  $\beta$ .
Figure 15: SEGA offers strong control over the semantic space and can gradually perform edits at the desired strength. All images generated from the same initial noise latent using the prompt 'a crowded boulevard'. Editing prompts denoted in blue and are gradually increased in strength from left to right and top to bottom. (Best viewed in color)
D Further Examples of Semantic Space's Properties
Here, we provide further examples of the properties of the semantic space created by SEGA (cf. Sec. 4). First, we revisit the uniqueness of guidance vectors. As shown in Fig. 3b these vectors may be calculated once and subsequently applied to other prompts. However, this is limited to the same initial noise latent and requires a similar image composition. We depict this difference in Fig. 8 where the guidance vector for 'glasses' is successfully applied to three images and fails to add glasses on three other images. However, the unsuccessfully edited images on the right retain the same image fidelity showcasing that SEGA's guidance vectors do not introduce artifacts. Nonetheless, SEGA can easily incorporate glasses into these images by computing a dedicated guidance vector (cf. Sec. 4).
Next, we demonstrate that multiple concept vectors scale monotonically and independent of each other. Meaning that in a multi-conditioning scenario the magnitude of each concept can be targeted independently. We depict an example in Fig. 15. Starting from the original image in the top left corner, we remove one concept from the image following the x- or y-axis. In both cases, SEGA monotonically reduces the number of people or trees until the respective concept is removed entirely. Again, the rest of the image remains fairly consistent, especially the concept that is not targeted. Going even further, a similar level of control is also possible with an arbitrary mixture of applied concepts. Let us consider the second row of Figure 15, for example. The number of trees is kept at a medium level in that the row of trees on the left side of each image is always removed, but the larger tree on the right remains. While keeping the number of trees stable, we can still gradually remove the crowd at the same time.
E User Study
We subsequently provide further details on the empirical evaluation presented in Sec. 5. We attempted to label the generated images using a classifier trained on CelebA. However, despite the model achieving over  $95\%$  accuracy on the CelebA validation set, the accuracy for the synthetic images was
Figure 16: Examples of improved visual quality through SEGA editing. Uncanny artifacts in the original images (left) are attenuated or removed by the additional guidance in image regions edited using SEGA (right). The used edit is. 'smiling'.
for some classes below  $20\%$ . Consequently, we opted to rely on human evaluators instead. For each of the 250 generated images we collected labels for all 10 attributes. Questions were posed in the form
Is the person wearing glasses?
The three answer options were:
Yes
No
Cannot Tell
or in the case of the gender attribute:
Male
Female
- Cannot Tell
Each user was tasked with labeling a batch of 28 image/attribute pairs; 25 out of those were randomly sampled from our generated images and each batch contained 3 hand-selected images from CelebA as sanity check. If users labeled the 3 CelebA images incorrectly the batch was discarded and added back to the task pool. Each images/attribute combination was labeled by 3 different annotators resulting in annotator consensus if at least 2 selected the same label.
Figure 17: Robustness of SEGA across multiple tasks. Here style transfer and image composition are easily combined and executed simultaneously. The resulting image satisfies both tasks. All images generated using the prompt 'a castle next to a river'. Top and bottom row use different seeds. (Best viewed in color)
To conduct our study we relied on Amazon Mechanical Turk where we set the following qualification requirements for our users: HIT Approval Rate over  $95\%$  and at least 1000 HITs approved. Annotators were fairly compensated according to Amazon MTurk guidelines. Users were paid $0.70 for a batch of 28 images at an average of roughly 5 minutes needed for the assignment.
In Fig. 16 we show examples of SEGA edits removing artifacts and improving overall uncanny image feel. Consequently, the evaluated image fidelity of the edited images is higher than of the originally generated ones.
F SEGA on various Architectures
As highlighted in the main body of this paper, SEGA can be integrated into any generative model using classifier-free guidance. Here we show further qualitative examples on various other architectures beyond Stable Diffusion. We implemented SEGA for two additional models: 1) Paella [26] which operates on a compressed and quantized latent space with iterative sampling over the distribution of predicted tokens per latent dimension. 2) IF  $^6$  a publicly available pixel-level diffusion model based on the Imagen architecture [28].
G Comparisons to related methods
Subsequently, we compare SEGA to other methods for image editing.
G.1 Direct Comparisons
We first compare against related methods for diffusion that allow for direct comparisons, i.e. the same seed and prompt will generate the exact same original image. All of these techniques have dedicated failure modes, i.e. use cases that these methods cannot accomplish. SEGA on the other behaves more robustly and successfully performs the desired edits in all scenarios.
G.1.1 Composable Diffusion
We first compare against Composable Diffusion [16]. In Fig 19 we can see that Composable Diffusion is also capable of conditioning on multiple concepts. However it makes more substantial changes than SEGA and produces artifacts when used with 3 or more concepts. Additionally, CompDiff performs significantly worse on image composition as shown in Fig. 20. For multiple of the applied
$^{6}$ https://github.com/deep-floyd/IF
Figure 18: Probing the diffusion model for learned concepts. Guiding towards or away from more complex concepts may reveal the underlying representation of the concept in the model. All images generated using the prompt 'a picture of a car'. (Best viewed in color)
Figure 19: Comparison of multiple edits being performed simultaneously for various semantic diffusion approaches.
Figure 20: Comparison of SEGA against Composable Diffusion [16] on image composition. SEGA significantly outperforms Composable Diffusion in terms of truthfulness to the original image and combination of multiple concepts.
and combined concepts the original image is no longer recognizable and SEGA also outperforms in terms of truthfulness of the targeted concepts and combinations thereof.
G.1.2 Prompt2Prompt
Prompt2Prompt [8] generally produces edits of similar quality as SEGA. However, Prompt2Prompt is limited to one editing prompt. This leads to the some concepts being ignored when aiming to edit multiple ones simultaneously as shown in Fig 19.
G.1.3 Disentanglement
The Disentanglement proposed by Wu et al. [35] performs similarly well to SEGA in applying multiple edits to faces (Fig 19). However, their method fails to realize small changes in an image as shown in Fig. 21. SEGA, on the other hand, faithfully executes the desired edit operation.
Figure 21: Comparison of SEGA against the approach of Wu et al. [35] for small edits. The Disentanglement makes no significant changes to the image, whereas SEGA faithfully executes the desired edit operation.
Figure 22: Comparison of SEGA against Negative Prompting. SEGA allows for dedicated control over the strength of the applied negative concept. Top row shows partial removal of the targeted concept with SEGA and middle row complete removal. Additionally, negative prompting breaks when used with multiple concepts simultaneously.
Additionally, the Disentanglement approach requires the optimization of a linear transformation vector, whereas SEGA can be applied ad hoc.
G.1.4 Negative Prompting
SEGA offers multiple benefits over negative prompting as shown in Fig 22. 1) Negative prompting offers no control over the strength of the negative guidance term, whereas SEGA provides a dedicated semantic guidance scale. 2) This often leads to unnecessarily large changes to the image when using negative prompts with the image oftentimes being completely different. With SEGA, however, changes can be kept to a minimum. 3) Negative prompting does not allow for removal of separate concepts, instead multiple concepts have to be provided in a single prompt. This may lead to one of the concepts not being removed in the generated image. In contrast, SEGA provides one guidance term with dedicated hyper parameters for each concept.
Figure 23: Comparison of SEGA against StyleCLIP.
Figure 24: Comparison of SEGA against StyleGAN-NADA.
G.2 Related Approaches
We also compare against two StyleGAN based approaches for semantic editing. Notably, we had to reconstruct the original image with StyleGAN which results in differences in the un-edited image.
G.2.1 Style-CLIP
SEGA achieves edits of similar fidelity as StyleCLIP [22] as shown in Fig. 23. However, SEGA can be applied ad hoc and does not require the time consuming mapping of a prompt against a style space attribute as need for StyleCLIP. Furthermore SEGA works on a powerful diffusion model capable of generating diverse images beyond one domain, contrary to a dedicated StyleGAN for faces.
G.2.2 StyleGAN-NADA
SEGA achieves edits of similar fidelity as StyleGAN-NADA [7] as shown in Fig. 24. However, SEGA can be computed on the fly and requires no tuning of the generative model as needed for StyleGAN-NADA. Furthermore SEGA works on a powerful diffusion model capable of generating diverse images beyond one domain, contrary to a dedicated StyleGAN for faces.
H Further Qualitative Evaluation
Lastly, we provide further qualitative examples further highlighting the capabilities and use cases of SEGA.
The isolation of concept vectors (cf. Sec. 4) not only allows for multiple simultaneous edits of the same task but also arbitrary combinations. In Fig. 17 we simultaneously change the image composition and change the style. The resulting image is a faithful interpolation of both changes applied individually. This further highlights both the robustness of SEGA guidance vectors as well as their isolation.
(a) Examples of image editing with prompt 'an image of a car'. All images are generated from the same noise latent and text prompts. SEGA performs high quality local and global edits with minimal changes to other aspects of the image.
(b) Optimization of artistic conception. Guidance conditioned on average embedding of prompt targeting one type of imagery. The resulting images are of high quality, reflect the targeted artistic style, while staying true to the original composition.
Figure 25: Qualitative Examples of Semantic Guidance on various tasks and domains. (Best viewed in color)
In Fig. 25a, we can see various edits being performed on an image of a car. This showcases the range of potential changes feasible with SEGA, which include color changes, altering the vehicle type or surrounding scenery. In each scenario, the guidance vector inferred using SEGA accurately targets the respective image region and performs changes faithful to the editing prompt. All while making next to no changes to the irrelevant image portions.
An additional benefit of semantic guidance directly on noise estimates is its independence on the modality of the concept description. In the case of SD, the modality happens to be text in natural language, but SEGA can be applied to any conditional diffusion model. We explore this idea further with the goal of optimizing the overall artistic conception of generated images. Instead of defining the target concept with one text prompt, we directly use an abstract embedding, representing a particular type of imagery. To that end, we collected prompts known to produce high-quality results for five different types of images in portrait photography, animation, concept art, character design, and modern architecture. The conditioning embedding for one style is calculated as the average over the embeddings of all collected prompts. Exemplary outputs are depicted in Fig. 25b. The results are of high quality and stay close to the original image but accurately reflect the targeted artistic direction beyond a single text prompt.
SEGA also contributes in uncovering how the underlying DM "interprets" more complex concepts and gives further insight into learned representations. We perform this kind of probing of the model in Fig. 18. For example, adding the concept 'carbon emissions' to the generated car produces a seemingly much older vehicle with a presumably larger carbon footprint. Similarly, reducing 'safety' yields a convertible with no roof and likely increased horsepower. Both these interpretations of the provided concepts with respect to cars are logically sound and provide valuable insights into the learned concepts of DMs. These experiments suggest a deeper natural language and image "understanding" that go beyond descriptive captions of images.
<sup>7</sup>Prompts taken from https://mpost.io/best-100-prompts