SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Uniqueness. Guidance vectors  $\gamma$  of one concept are unique and can thus be calculated once and subsequently applied to other images. Fig. 3b shows an example for which we computed the semantic guidance for 'glasses' on the left-most image and simply added the vector in the diffusion process of other prompts. All faces are generated wearing glasses without a respective  $\epsilon$ -estimate required. This even covers significant domain shifts, as seen in the one switching from photo-realism to drawings.
However, the transfer is limited to the same initial seed, as  $\epsilon$ -estimates change significantly with diverging initial noise latents. Furthermore, more extensive changes to the image composition, such as the one from human faces to animals or inanimate objects, require a separate calculation of the guidance vector. Nonetheless, SEGA introduces no visible artifacts to the resulting images.
Monotonicity. The magnitude of a semantic concept in an image scales monotonically with the strength of the semantic guidance vector. In Fig. 3c, we can observe the effect of increasing the strength of semantic guidance  $s_e$ . Both for positive and negative guidance, the change in scale correlates with the strength of the smile or frown. Consequently, any changes to a generated image can be steered intuitively using only the semantic guidance scale  $s_e$  and warm-up period  $\delta$ . This level of control over the generation process is also applicable to multiple concepts with arbitrary combinations of the desired strength of the edit per concept.
Isolation. Different concepts are largely isolated because each concept vector requires only a fraction of the total noise estimate. Meaning that different vectors do not interfere with each other. Thus, multiple concepts can be applied to the same image simultaneously, as shown in Fig. 4. We can see, for example, that the glasses which were added first remain unchanged with subsequently added edits. We can utilize this behavior to perform more complex changes, best expressed using multiple concepts. One example is the change of gender by simultaneously removing the 'male' concept and adding the 'female' one (cf. Figs. 5 and 6).
Next, we present exhaustive evaluation of semantic guidance on empirical benchmarks, as well as additional qualitative tasks. Furthermore, we report user preferences on direct comparisons with existing methods [16, 8, 35]. Our experiments refute claims of previous research, by demonstrating
Figure 4: Successive combination of concepts. From left to right an new concept is added each image. Concepts do not interfere with each other and only change the relevant portion of the image. (Best viewed in color)
Figure 5: Examples from our empirical evaluation benchmark, showing the 10 attributes edited with SEGA. Original and edited images are evaluated by human users one feature at a time. (Best viewed in color)
the suitability of noise estimates for semantic control [15] and its capability for small, subtle changes [35]. The comparison to other methods diffusion-based approaches [8, 35, 16] indicate SEGA's improved robustness and capability to cover a wider range of use cases.
Empirical Results. First, we performed an extensive empirical evaluation on human faces and respective attributes. This setting is inspired by the CelebA dataset [17] and marks a well-established benchmark for semantic changes in image generation. We generated 250 images with unique seeds using the prompt 'an image of the face of a random person' and manipulated ten facial attributes. These attributes are a subset of the CelebA labels. All attributes and respective examples of the corresponding manipulation using SEGA are depicted in Fig. 5. In addition to additive image edits, we evaluated negative guidance of these attributes as well as two combinations of four simultaneous edits, as shown in Fig. 4. Therefore, our empirical evaluation spans 21 attribute changes and combinations in total with all images being generated using the Stable Diffusion implementation.
We evaluated the generated images with a user study and provide more details on its implementation in App. E. The results are shown in Tab. 1. For positive guidance, SEGA faithfully adds the target concept to the image on average in  $95\%$  of the cases, with the majority of attributes exceeding  $97\%$ . We further manually investigated the two outliers, 'Bald' and 'Bangs'. We assume that many of the non-native English-speaking annotators were not familiar with the term 'bangs' itself. This assumption is based on correspondence with some of the workers, and the conspicuously low rate of annotator consensus. Consequently, the numbers for 'bangs' should be taken with a grain of salt. For baldness, we found that long hair often makes up a large portion of a portrait and thus requires more substantial changes to the image. Consequently, such edits require stronger hyperparameters than those chosen for this study. Furthermore, we observe negative guidance to remove existing attributes from an image to work similarly well. It is worth pointing out that the guidance away from 'beard', 'bald' or 'gray hair' usually resulted in a substantial reduction of the respective feature, but failed to remove it entirely for  $\sim 10\%$  of the images. Again, this suggests that the hyperparameters were probably not strong enough.
Lastly, we looked into the simultaneous guidance of multiple concepts at once. The results in Tab. 2 empirically demonstrate the isolation of semantic guidance vectors. The per-attribute success rate remains similar for four instead of one distinct edit concept, suggesting no interference of guidance vectors. Consequently, the success of multiple edits only depends on the joint probability of the individual concepts. In comparison, if only two out of four applied concepts were interfering with each other to be mutually exclusive, the success rate of such a combination would always be  $0\%$ . Contrary to that, we successfully apply concurrent concepts in up to  $91\%$  of generated images.
Additionally, we investigated any potential influence of manipulations with SEGA on overall image quality. Utilizing the facial images from the previous experiment, we calculated FID scores against a
Table 1: Empirical results of our user study conducted on face attributes. Sample sizes result from the portion of the 250 original images that did not contain the target attribute. Annotator consensus refers to the percentage of images for which the majority of annotators agreed on a label. Success rate is reported on those images with annotator consensus.
Attribute	#	Consensus(%)	Success(%)
Pos. Guidance	Gender	241	99.2	100.0
Glasses	243	99.6	100.0
Smile	146	100.0	99.3
Bald	220	91.2	82.1
Beard	135	97.8	97.0
Hat	210	99.0	99.0
Curls	173	95.4	97.0
Makeup	197	99.5	99.0
Gray hair	165	97.6	91.2
Bangs	192	86.2	82.7
Overall	1922	96.5	95.0
Neg. Guidance	No Glasses	6	98.9	100.0
No Smile	93	100.0	94.4
No Bald	18	94.8	83.3
No Beard	111	100.0	89.9
No Hat	31	100.0	90.3
No Curls	50	98.0	98.0
No Makeup	21	100.0	95.2
No Gray Hair	52	94.6	82.7
No Bangs	38	95.0	98.5
Overall	420	91.9	92.1
Table 2: Results of user study on simultaneous combination of face attributes. Sample sizes result from the portion of the 250 original images that did not contain any target attribute. Annotator consensus refers to the percentage of images for which the annotator's majority agreed on a label. Success rates for combinations on any combination of  $x$  attributes with per-attribute scores are reflecting the isolated success of that edit. Nonetheless, all scores are shown for images with all 4 edit concepts applied simultaneously.
Attribute	#	Consensus(%)	Success(%)
Combination 1	≥ 1 Attr.	100.0
≥ 2 Attr.	55	98.2
≥ 3 Attr.
All 4 Attr.	90.7
Glasses	100.0
Smile	55	100.0
Curls	96.4
Beard	98.2
Combined 2	45	100.0
≥ 1 Attr.	100.0
≥ 2 Attr.	100.0
≥ 3 Attr.	100.0
All 4 Attr.	75.6
No Smile	97.8
Makeup	45	100.0
Hat	97.6
Female	88.6
Total	100.0
reference dataset of FFHQ [11]. FID scores for the original, unedited images are remarkable bad at 117.73, which can be attributed to small artifacts often present in facial images generated by Stable Diffusion. Editing images with SEGA significantly improved their quality resulting in an FID score on FFHQ of 59.86. Upon further investigation, we observed that the additional guidance signal for a dedicated portion of the faces frequently removed uncanny artifacts, resulting in overall better quality.
Further, we empirically evaluated SEGA on the significantly different task of counteracting inappropriate degeneration [3, 4, 29]. For latent DMs, Schramowski et al. [29] demonstrated that an additional guidance term may be used to mitigate the generation of inappropriate images. We demonstrate that SEGA's flexibility enables it to now perform this task on any diffusion architecture, both latent and pixel-based. To that end, we evaluated Stable Diffusion v1.5 and DeepFloyd-IF on the inappropriate-image-prompts (I2P) benchmark [29]. Subsequently, we suppressed inappropriate content using SEGA to guide the generation away from the inappropriate concepts as specified by Safe Latent Diffusion (SLD) [29]. We empirically chose the SEGA parameters to produce results similar to the 'Strong' configuration of SLD. The results in Tab. 3 demonstrate that SEGA performs strong mitigation at inference for both architectures further highlighting the capabilities and versatility of the approach.
Comparisons. In addition to the isolated evaluation of SEGA's capabilities, we conducted randomized user studies directly comparing SEGA with related methods. This comparison investigates the performance of Composable Diffusion [16], Prompt2Prompt [8], Disentanglement [35], and SEGA on tasks from four types of manipulation categories, reflecting a broad range of operations. Specifically, we considered 1) composition of multiple edits, 2) minor changes, 3) style transfer and 4) removal of specific objects from a scene. For each method and task, we selected the best-performing hyperparameters on a set of seeds  $< 100$  and subsequently generated non-cherry-picked edits on a fixed test set (i.e., first applicable seeds  $>= 100$ ). As first evaluation step, we conducted a user study to assess the success of an edit based on the presence/absence of the target attribute(s) as shown in Tab. 4. The comparatively low scores on removal tasks result from the methods often reducing the presence of the targeted objects but not eliminating all of them in some cases. However, this peculiarity of the evaluation affects all methods similarly and does not influence the comparability of capabilities. The results demonstrate that SEGA clearly outperforms Prompt2Prompt and Disentanglement on all
Table 3: Results of Stable Diffusion and IF on the I2P benchmark [29]. Values reflect the probability of generating inappropriate content (the lower the better) with respect to the joint Q16/NudeNet classifier proposed by Safe Latent Diffusion[29]. Subscript values denote the expected maximum inappropriateness over 25 prompts (the lower the better). For both architectures SEGA performs strong mitigation at inference.
Harassment	Hate	Illegal Activity	Self-Harm	Sexual	Shocking	Violence	Overall
SD	0.320.92	0.390.96	0.350.94	0.400.97	0.290.86	0.511.00	0.420.98	0.380.97
SD w/ SEGA	0.120.66	0.150.75	0.090.57	0.070.56	0.040.36	0.190.84	0.140.75	0.110.68
IF	0.350.98	0.461.00	0.400.99	0.401.00	0.220.91	0.491.00	0.431.00	0.380.99
IF w/ SEGA	0.160.84	0.200.88	0.150.84	0.130.81	0.060.58	0.210.92	0.180.89	0.150.84
Table 4: User study results of success rates for different manipulation types and methods. Overall SEGA outperforms all compared techniques and specifically improves on tasks with multiple edits and small changes. Success rate is reported on those images where the majority of annotator agree on a label.
Method	Multi-Conditioning (%)	Minor Changes (%)	Style Transfer (%)	Concept Removal (%)	Overall (%)
Composable Diffusion[16]	35.00	72.00○	100.0●	35.00●	60.50○
Prompt2Prompt[8]	35.00	68.00	65.00	5.00	43.25
Disentanglement[35]	35.00	65.38	65.00	0.00	41.35
SEGA (Ours)	80.00●	90.91●	90.00○	30.00○	72.72●
examined editing tasks. Compared to Composable Diffusion, SEGA again has significantly higher success rates for multi-conditioning and minor changes while achieving comparable performance for style transfer and object removal.
Beyond the simple success rate of edits, we evaluated the faithfulness to the original image composition. To that end, users considered pair-wise comparisons of samples that both methods edited successfully and assessed the similarity with the original image. Importantly, users strongly prefer SEGA results over Composable Diffusion for  $83.33\%$  (vs.  $13.33\%$ ) of samples. These two studies highlight that SEGA is generally preferred in terms of its edit capabilities and perceived fidelity over other methods. We present examples and further details on comparisons to related diffusion and StyleGAN techniques in App. G.
Qualitative Results. In addition to the empirical evaluation, we present qualitative examples on other domains and tasks. We show further examples in higher resolution in the Appendix. Overall, this highlights the versatility of SEGA since it allows interaction with any of the abundant number of concepts diffusion models are capable of generating in the first place. We performed a diverse set of style transfers, as shown in Fig. 6. SEGA faithfully applies the styles of famous artists, as well as artistic epochs and drawing techniques. In this case, the entirety of the image has to be changed while keeping the image composition the same. Consequently, we observed that alterations to the entire output—as in style transfer—require a slightly lower threshold of  $\lambda \approx 0.9$ . Nonetheless, this still means that  $10\%$  of the  $\epsilon$ -space is sufficient to change the entire style of an image. Fig. 6 also includes a comparison between outputs produced by SEGA with those from simple extensions to the prompt text. Changing the prompt also significantly alters the image composition. These results further highlight the advantages of semantic control, which allows versatile and yet robust changes.
6 Broader Impact on Society
Recent developments in text-to-image models [25, 21, 28] have the potential for a far-reaching impact on society, both positive and negative, when deployed in applications such as image generation, image editing, or search engines. Previous research [3, 29] described many potential negative societal implications that may arise due to the careless use of such large-scale generative models. Many of these problems can be attributed to the noisy, large-scale datasets these models rely on. Since recent text-to-image models, such as SD, are trained on web-crawled data containing inappropriate content [30], they are no exception to this issue. Specifically, current versions of SD show signs of inappropriate degeneration [29]. While Schramowski et al. [29] utilize the model's notion of inappropriateness to steer the model away from generating related content, it is noteworthy that we introduce an approach that could also be used to guide image generation toward inappropriate material. The limitations of Stable Diffusion also effect our editing of perceived gender which may
Figure 6: Qualitative examples using SEGA. Top row demonstrates that the approach is architecture agnostic showing examples for for Stable Diffusion<sup>3</sup>, Paella [26] and Deepfloyd-IF<sup>4</sup>. Below we show style transfer using SEGA. All images are generated from the same noise latent using the text prompt 'a house at a lake'. SEGA easily applies the characteristics of a specific artist, epoch or drawing/imaging technique to the original image while preserving the overall composition. In contrast, appending the prompt with a style instruction results in images that significantly change the composition. (Best viewed in color)
exhibit potential biases. Since Stable Diffusion's learned representation of gender are limited, we restrict ourselves to binary labels, although gender expression cannot be ascribed to two distinct categories. Furthermore, does Stable Diffusion contain severe biases in its attribution of gender and correlated features that may further reinforce pre-existing stereotypes. Since editing methods in general, and SEGA in particular, rely on these representations of the underlying model, they will inevitably inherit them for related editing operations. However, on the positive side, SEGA has the potential to mitigate bias. As demonstrated by Nichol et al.[21], removing data from the training set has adverse effects, e.g., on a model's generalization ability. In contrast, SEGA works at inference promoting fairness in the outcome. Therefore, we advocate for further research in this direction.
Another frequently voiced point of criticism is the notion that generative models like SD are replacing human artists and illustrators. Indeed, certain jobs in the creative industry are already threatened by generative image models. Furthermore, the collection of training data for (commercial) applications is often ethically questionable, building on people's artwork without their explicit consent. Nonetheless, we believe SEGA to be promoting creating artwork in an interactive process that requires substantial amount of iterative human feedback.
We introduced semantic guidance (SEGA) for diffusion models. SEGA facilitates interaction with arbitrary concepts during image generation. The approach requires no additional training, no extensions to the architecture, no external guidance, is calculated during the existing generation process and compatible with any diffusion architecture. The concept vectors identified with SEGA are robust, isolated, can be combined arbitrarily, and scale monotonically. We evaluated SEGA on a variety of tasks and domains, highlighting—among others—sophisticated image composition and editing capabilities.
Our findings are highly relevant to the debate on disentangling models' latent spaces. So far, disentanglement as a property has been actively pursued [12]. However, it is usually not a necessary quality in itself but a means to an end to easily interact with semantic concepts. We demonstrated that this level of control is feasible without disentanglement and motivate research in this direction.
Additionally, we see several other exciting avenues for future work. For one, it is interesting to investigate further how concepts are represented in the latent space of DMs and how to quantify them. Similarly, SEGA could be extended to allow for even more complex manipulations, such as targeting different objects separately. More importantly, automatically detecting concepts could provide novel insights and toolsets to mitigate biases, as well as enacting privacy concerns of real people memorized by the model.
Acknowledgments This research has benefited from the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) cluster projects "The Third Wave of AI" and hessian.AI, from the German Center for Artificial Intelligence (DFKI) project "SAINT", the Federal Ministry of Education and Research (BMBF) project KISTRA (reference no. 13N15343), as well as from the joint ATHENE project of the HMWK and the BMBF "AVSV". Further, we thank Amy Zhu for her assistance in conducting user studies for this work.
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics, 2023.
[2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv:2211.01324, 2022.
[3] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2022.
[4] Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), 2021.
[5] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[6] Kevin Frans, Lisa B. Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[7] Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics, 2022.
[8] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[10] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022.
[11] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[12] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[13] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[14] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv:2107.00630, 2021.