SEGA: Instructing Text-to-Image Models using Semantic Guidance
Authors: Manuel Brack
Felix Friedrich
Dominik Hintersdorf
Lukas Struppek
Patrick Schramowski
Kristian Kersting
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion
text-to-image
generation
semantics

Abstract: Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user’s intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, it allows for subtle and extensive edits, composition and style changes, and optimizing the overall artistic conception. We demonstrate SEGA’s effectiveness on both latent and pixel-based diffusion models such as Stable Diffusion, Paella, and DeepFloyd-IF using a variety of tasks, thus providing strong evidence for its versatility and flexibility.

SEGA: Instructing Text-to-lmage Models using Semantic Guidance
Manuel Brack
Felix Friedrich
Dominik Hintersdorf? Lukas Struppek2
Patrick Schramowski Kristian Kersting 12,3,55 German Research Center for Artificial Intelligence (DFKI), Computer Science Department, TU Darmstadt 3 Hessian.AI,
*LAION, "Centre for Cognitive Science, TU Darmstadt brackêcs. tu-darmstadt.co
Abstract
Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achiev- ing one-shot generation that aligns with the user's intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions.
This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, allows for subtle and extensive edits,
changes in composition and style, as well as optimizing the overall artistic concep- tion. We demonstrate SEGA'S 's effectiveness models such as Stable Diffusion, Paella and DeepFloyd-IF using variety of tasks, thus providing strong evidence for its versatility, flexibility and improvements over existing method!
both latent and pixel-based diffusion
Introduction
The recent popularity of toxt-to--mage diffusion models (DMs) 28 25 27 can largely be attributed to their versatility, expressiveness, and and-most most importantly the intuitive interface they provide to users. The generation's intent can easily be expressed natural language, with the model producing faithful interpretations of text prompt. Despite the impressive capabilities of these models, the initially generated images are rarely of high quality. Accordingly, a human user will likely be unsatisfied with certain aspects of the initial image, which they will attempt to improve over multiple iterations. Unfortunately, the diffusion process is rather fragile as small changes to the input prompt lead to entirely different images. Consequently, fine-grained semantic control over the generation process is necessary, which should be as easy and versatile to use as the initial generation.
Previous attempts to influence dedicated concepts during the generation process require additional segmentation masks, extensions to the architecture, model fine-tuning, or embedding optimization [ 35]. While these techniques produce satisfactory results, they disrupt the fast, exploratory workflow that is the strong suit of diffusion models in the first place. We propose Semantic Guidance (SEGA) to uncover and interact with semantic directions inherent to the model. SEGA requires no additional training, no extensions to the architecture, nor external guidance and calculated within single forward pass. We demonstrate that this semantic control can be inferred from simple textual descriptions using the model's noise estimate alone. With this, we also refute previous research
'Imllementation available in diffusers: http://wwnggh.ooooooodooeeeeeess
diffusion
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original
king
male
female
king "malee "eenale"
sonnos
Luidarce
derougt
of king
(a) A (latent) diffusion process inherently organizes concepts and learns implicitly relationships between them, although there is supervision.
(b) Guidance arithmetic: Guiding the image 'a portrait of king' (left) using "king" - "'mal' + 'female' results in image of a "queen' (right).
Figure 1: Semantic guidance (SEGA) applied to the image 'a portrait of a king" (Best viewed in color)
claiming these estimates to be unsuitable for semantic control 15 The guidance directions uncovered with SEGA are robust, scale monotonically, and are largely isolated. This enables simultaneous applications subtle edits to images, changes in composition and style, as well as optimizing the artistic conception. Furthermore, SEGA allows for probing the latent space of diffusion models to gain insights into how abstract concepts are represented by the model and how their interpretation reflects the generated image. Additionally, SEGA is architecture-agnostic and compatible with various generative models, including latent [7 26 and pixel-based diffusion (28).
In this paper, establish the methodical benefits of SEGA and demonstrate that this intuitive, lightweight approach offers sophisticated semantic control over image generations. Specifically, contribute by (i) devising formal definition of Semantic Guidance and discussing the numer- ical intuition of the corresponding semantic space, (ii) demonstrating the robustness, uniqueness, monotonicity, and isolation of semantic vectors, (iii) providing exhaustive empirical evaluation of SEGA's 's semantic control, and (iv) showcasing the benefits of SEGA over related methods.
Background
Semantic Dimensions. Research on expressive semantic vectors that allow for meaningful interpola- tion and arithmetic pre-date generative diffusion models. Addition and subtraction text embeddings such as word2vec 18
19] have been shown to reflect semantic and linguistic relationships in natural One of the most prominent examples is that the vector representation of 'King male + female' is very close to 'Queen'. SEGA enables similar arithmetic for image generation with diffusion (cf. Fig. Tbp. StyleGANs [2 also contains inherent semantic dimensions that can be utilized during generation. For example, Patashnik et al. (22) combined these models with CLIP (24 to offer limited textual control over generated attributes. However, training StyleGANs at scale with subsequent fine-tuning is notoriously fragile due to the challenging balance between reconstruction and adversarial loss. Yet, large-scale pre-training the base of flexible and capable generative models 23
language 200 34
Image Diffusion. Recently, large-scale, lext-guided DMs have enabled more versatile approach for image generation [28] 25 (2]. Especially latent diffusion models (27] have been gaining much attention. These models perform the diffusion process compressed space perceptually equivalent to the image space. For one, this approach reduces computational requirements. Additionally, the
latent representations can be utilized for other downstream applications 6 17
Image Editing. While these models produce astonishing, high-quality images, fine- grained control over this process remains challenging. Minor changes to the text prompt often lead to entirely different images. One approach to tackle this issue is inpainting, where the user provides additionally semantic masks to restrict changes to certain areas of the image [| 21) Other methods involve computationally expensive fine-tuning -eeiinnng of the model to condition it on the source image before applying edits [3] 33 In contrast, SEGA performs edits the relevant image regions through text descriptions alone and requires no tuning.
Semantic Control. Other works have explored more semantically grounded approaches for interact- ing with image generation. Compp-to-Prompp utilizes the semanties of the model's cross-attention layers that attribute pixels to tokens from the text prompt [8]. Dedicated operations on the cross-

---

attention maps enable various changes to the generated image. On the other hand, SEGA does not require token-based conditioning and allows for combinations multiple semantic changes. Wu et al. of concepts for DMs using linear combinations of text embeddings. However, for each text prompt and target concept, dedicated combination must be inferred through optimization. Moreover, the approach only works for more substantial changes to an image and fails for small edits. SEGA, in contrast, is capable of performing such edits without optimization.
35 studied the disentanglemett
Eatse-Esttmate Manipulation. Our work is closely related to previous research working directly on the noise estimates of DMs. Liu et al. [16] combine multiple estimates to facilitate changes in image composition. However, more subtle semantic changes to an image remain unfeasible with this method. In fact, Kwon et al. [15] argue that the noise-estimate space of DMs is unsuited for semantic manipulation of the image. Instead, they use a learned mapping function on changes to the bottleneck of the underlying U-Net. This approach enables various manipulations that preserve the original image quality. However, it does not allow for arbitrary spontaneous edits of the image, as each editing concept requires minutes of training. SEGA, in comparison, requires no extension to the architecture and produces semantic vectors ad-hoe for any textual prompt. Lastly, Safe Latent Diffusion (SLD) uses targeted manipulation of the noise estimate to suppress the generation of inappropriate content [29]. Instead of arbitrary changes to image, SLD prevents one dedicated concept from being generated. Additionally, SLD is complex, and the hyperparameter formulation can be improved through a deeper understanding of the numerical properties of DMs' noise estimate space.
3
Semantic Guidance
Let us now devise Semantic Guidance for diffusion models.
3.1 Guided Diffusion
The first step towards SEGA is guided diffusion. Specifically, diffusion models (DM) iteratively denoise Gaussian distributed variable to produce samples of a learned data distribution. For text-to- image generation, the model is conditioned on a text prompt p and guided toward an image faithful to that prompt. The training objective of DM in can be written
Ex,e,,e.t [wx | (axx + Wec, cp) - x1
where (x, cp) is conditioned on text prompt p. is drawn from a uniform distribution ~ ([0, 1)). c sampled from a Gaussian Consequently, the DM is trained to denoise := x + yielding x with the squared error loss. At inference, the DM is sampled using the prediction of x (z, Ee), with &g as described below.
NO I), and wy, Wh, Or influence the image fidelity depending on t.
Classifier-free guidance 10 is a conditioning method using a purely generative diffusion model, eliminating the need for additional pre-trained classifier. During training, the text conditioning Cp drops randomly with a fixed probability, resulting in a joint model for unconditional and conditional objectives. During inference, the score estimates for the x-prediction are adjusted so that:
Eo(Zi, cp) co(z,) + (e Zz, cp) ce(z))
(2)
with guidance scale s and eg defining the noise estimate with parameters 0. Intuitively, the uncondi- tioned e- e-prediction pushed in the direction of the conditioned one, with s, determining the extent of the adjustment.
3.2 Semantic Guidance on Concepts
We introduce SEGA to influence the diffusion process along several directions. To this end, we substantially extend the principles introduced in classifier-free guidance by solely interacting with the concepts already present in the model's latent space. Therefore, SEGA requires no additional training, no extensions the architecture, and no external guidance. Instead, it is calculated during the existing diffusion iteration. More specifically, SEGA uses multiple textual descriptions e;, representing the given target concepts of the generated image, in addition the text prompt p.
Intuition. The overall idea of SEGA best explained using 2D abstraction of the high dimensional e-space, as shown in Fig. I Intuitively, we can understand the space as a composition of arbitrary
3

---

sub-spaces representing semantic concepts. Let us consider the example of generating an image of a king. The unconditioned noise estimate (black dot) starts at some random point in the e-space without semantic grounding. The guidance corresponding to the prompt "a portrait of a king" represents vector (blue vector) moving us into portion of 6-space where the concepts 'male' and royal overlap, resulting in an image of king. We can now further manipulate the generation process using SEGA. From the unconditioned starting point, we get the directions of 'male' and 'female' (orange/green lines) using estimates conditioned on the respective prompts. If we subtract this inferred 'male' direction from our prompt guidance and add the 'female' one, we now reach a point in the e-space at the intersection of the 'royal' and 'female' sub-spaces, i.., queen. This vector represents the final direction (red vector) resulting from semantic guidance.
Isolating Semanties in Diffusion. Next, we investigate the actual noise-estimate space of DMs on the example of Stable Diffusion (SD). This enables extracting semantic concepts from within that space and applying them during image generation.
Numerical values of e-estimates are generally Gaussian distributed. While the value in each di- mension of the latent vector can differ significantly between seeds, text prompts, and diffusion steps, the overall distribution always remains similar Gaussian distribution (cf. App. B). Using the arithmetic principles of classifier-free guidance, we can now identify those dimensions of latent vector encoding an arbitrary semantic concept. To that end, we calculate the noise estimate co co(Z,, (z), which is conditioned on a concept description e. We then take the difference between co(z,, Cc) and the unconditioned estimate (Z())) and scale it. Again, the numerical values of the resulting latent vector are Gaussian distributed, as shown in Fig. We will demonstrate that those latent dimensions falling into the upper and lower tail of the distribution alone encode the target concept. We empirically determined that using only 1-5% of the e-estimate's ' dimensions is suf- ficient to apply the desired changes to an image. Consequently, the resulting concept vectors are largely isolated; thus, multiple ones can be applied simultaneously without interference (ef. Sec. 4
We subsequently refer to the space these sparse noise-estimate vectors as semanlic space.
300
Semantic Difference
One Direction. Let us formally define the intuition for SEGA by starting with single direction, i.c., editing prompt. Again, we use three c- e-predictions to move the unconditioned score estimate co (2,) to- wards the prompt conditioned estimate co co cp) and simultancously away/towards the concept condi- tioned estimate e (Z,,, C,), depending on the editing direction. Formally, we compute Eg (Ze, Cp. C.)
Scaled Sem Difference
40
Guidance Values
Density
300
20
10
0.15
-0.10
-0.05
0.00
0.06
0.10
0.15
Numerical Value Noise Estimate
(o(2.) $g (co( cp) co(2)) + (2,, Ce)
(3)
Figure 2: Numerical intuition of semantic guidance. The difference between the concept-conde condpt-con and unconditioned estimates first scaled. Sub- sequently, the tail values represent the dimensions of the specified concept. Distribution plots calcu- lated using kernel-density estimates with Gaussian smoothing.
with the semantic guidance term y
7(z,, = μ(v: Se, A) c.)
(4)
where f applies an edit guidance scale s, element- wise, and v depends on the edit direction:
co ce) co(zz) if pos. guidance Vize, C.) 11 ( ce) co(z)) if neg. guidance
(5)
Thus, changing the guidance direction reflected by the direction between ca(),,c.) and eæ(2.).
The term i (Eq. 4 considers those dimensions of the prompt conditioned estimate relevant to the defined editing prompt e. To this end, μl takes the largest absolute values of the difference between the unconditioned and concept-condition the numerical distribution as defined by percentile threshold X. All values in the tails are scaled by an edit scaling factor 8c, with everything else being set to 0, such that
estimates. This corresponds to the upper and lower tail of
where |bb 2 nx (b+))
(( Sc, A)
(6)
otherwise
where 7)x (x) is the X-th percentile of v. Consequently, larger 8, increases SEGA'S effect.

---

SEGA can also be theoretically motivated in the mathematical background of DMs 32 14]. The isolated semantic guidance equation for the positive direction without the classifier- free guidance term can be written
To (z(z Ce) = (o(z)) + (a(co(z),cc) ce) co((z))
given Eqs. 4 Further, let assume an implicit classifier for p(ce|zz) x p(Re.lce), where p(z) is p(ze)
the marginal for the variance- pereeerreiine Markov process ((2[z) and xr ~ p(x) [1 Assuming
exact estimates c" (zz: ce) of p(z/ and e² e'(z)) of p(z)) the gradient of the resulting classifier can be written as Zz, dog.(cc.zz [t) = we "e( (Zz, Ce) [ -<=((c))) Using this implicit classifier for classifier guidance [5] results in noise estimate e" ("z,, C.) = (z, + wze" Ce) E (z))). This is fundamentally similar to SEGA as shown in Eq. 7 with l isolating the dimensions of the classifier signal with the largest absolute value (cf. Eq. 6). However, it should be noted that E (2,, ce) is profoundly different from Eo (Z,, Ge) as the latter's 's expressions are outputs of an unconstrained network and not the gradient of a classifier. Consequently, there performance of SEGA. Nevertheless, this derivation provides a solid theoretical foundation and we able demonstrate the effectiveness of SEGA empirically in Sec. 5
no guarantees [10] for the
To offer even more control over the diffusion process, we make two adjustments to the methodology presented above. We add warm-up parameter 5 that will apply guidance 7 after an initial warm-up period in the diffusion process, i.e., 7(z,, C.) : if <.. Naturally, higher values for lead less substantial adjustments of the generated image. If we aim to keep the overall image composition unchanged, selecting a sufficiently high ō ensures only altering fine-grained output details.
Furthermore, we add momentum term to the semantic guidance 7 to accelerate guidance over time steps for dimensions that are continuously guided in the same direction. Hence, Y/ is defined as:
(Ze, c.) = p(bb Se: (ze, ce) + Sm
(8)
with momentum scale 8m. [0. 1] and v being updated as
Vt+1 = Bm + (1 - Bm)
(9)
where 1/) =0 and 8m E [0, 1). Thus, larger 8. lead to less volatile changes in momentum. Momentum already built up during the warm-up period, even though " is not applied during these steps.
Beyond One Direction. Now, we are ready to move beyond using just one direction towards multiple concepts e and, in turn, combining multiple calculations of ?/.-
For all ej calculate y described above with each defining their own hyperparameter values X sy. The weighted sum of all oi results in
(z, Cos) = giri Ce.)
(10)
In order to account for different warm-up periods. Ji is defined 9: 0 8,. However, momentum is built up using all editing prompts and applied once all warm-up periods are completed, i.e., Vd, 6, I. We provide pseudo-code implementation of SEGA in App-|Alane more detailed intuition of each hyper-parameter along with visual ablations in App.
SEGA'S underlying methodology is architecture-agnosti and applicable any model employing classifier-free guidance. Consequently, we implemented SEGA for various generative models of different architectures and make code available online. If not stated otherwise the examples presented in the main body, are generated using our implementation based on SD v with images from other models and architectures included in App. F We note that SEGA easily be applied to real images using reconstruction techniques for diffusion models, which we regard as future work.
4 Properties of Semantic Space
With the fundamentals of semantic guidance established, we next investigate the properties of SEGA'S semantic space. addition to the following discussion, we present further examples in the Appendix.
5

---

(a) Robustness of guidance vectors. Results for guid- ing towards 'glasses' in various domains without spec- ifying how concept should incorporated.
(b) Uniqueness of guidance vectors. The vector for "glasses' is calculated once the blue-marked image and subsequently applied to other prompts (without colored border).
Decroasc 'ankt'
Orignal
Incrcase atsie"
(c) Monotonicity of guidance vectors. The guidance scale for 'smile' is semantically reflected in the images.
Figure 3: Robustness, uniqueness and monotonicity of SEGA guidance vectors, In a) and the top row depicts the unchanged image, bottom row depicts the ones guided towards 'glasses'. (Best viewed in color)
Robustness. SEGA behaves robustly for incorporating arbitrary concepts into the original image. In Fig. [3l we applied guidance for the concept 'glasses' to images from different domains. Notably, this prompt does not provide any context on how to incorporate the glasses into the given image and thus leaves room for interpretation. The depicted examples showcase how SEGA extracts best-effort integration of the target concept into the original image that is semantically grounded. This makes SEGA'S 's use easy and provides the same exploratory nature as the initial image generation.
Uniqueness. Guidance vectors of concept are unique and thus be calculated subsequently applied to other images. Fig. shoss an example for which we computed the semantic guidance for 'glasses' the left-most image and simply added the vector in the diffusion process of other prompts. All faces are generated wearing glasses without a respective c-estimate required. This even covers significant domain shifts, as seen in the one switching from photo-realism drawings.
and
However, the transfer is limited to the same initial seed, as e-estimates change significantly with diverging initial noise latents. Furthermore, as the one from human faces to animals or inanimate objects, require separate calculation of the guidance vector. Nonetheless, SEGA introduces no visible artifacts to the resulting images.
extensive changes the image composition, such
Monotonicity. The magnitude of semantic concept an image scales monotonically with the strength of the semantic guidance vector. In Fig. [3]] we can observe the effect of increasing the strength of semantic guidance Se- Both for positive and negative guidance, the change in scale correlates with the strength of the smile or frown. Consequently, any changes to generated image can steered intuitively using only the semantic guidance scale 8, and warm-up period ò. This level of control over the generation process is also applicable to multiple concepts with arbitrary combinations of the desired strength the edit per concept.
Isolation. Different concepts are largely isolated because each concept vector requires only a fraction of the total noise estimate. Meaning that different vectors do not interfere with each other. Thus, multiple concepts can be applied to the same image simultaneously, as shown in Fig. 4 We can see, for example, that the glasses which were added first remain unchanged with subsequently added edits. We can utilize this behavior to perform more complex changes, best expressed using multiple concepts. One example is the change of gender by simultaneously removing the 'male' concept and adding the 'female" one (cf. Figs. 5 and
5 Experimental Evaluation
Next, we present exhaustive evaluation of semantic guidance empirical benchmarks, as well as additional qualitative tasks. Furthermore, we report user preferences on direct comparisons with
existing methods [16 8 35 Our experiments refute claims of previous research, by demonstrating

---

Laulyy haie glooes
Curlyy hai closses
Cunly hair beardd
Original
Figure 4: Successive combination of concepts. From left to right concept is added each image. Concepts do not interfere with each other and only change the relevant portion of the image. (Best viewed in color)
Original
Female
Glasses
Beard
Curls
Makeup
Gray, Hair
Bungs
Figure 5: Examples from empirical evaluation benchmark, showing the 10 attributes edited with SEGA.
Original and edited images are evaluated by human users feature at time. (Best viewed color)
the suitability of noise estimates for semantic control 15 and its capability for small, subtle changes
335. The comparison to other methods diffusion- based approaches [ 35 16 indicate SEGA'S
improved robustness and capability to cover wider range of use
Empirical Results. First, we performed an extensive empirical evaluation on human faces and respective attributes. This setting is inspired by the CelebA dataset [17] and marks well-established benchmark for semantic changes in image generation. We generated 250 images with unique seeds using the prompt 'an image of the face of a random person' and manipulated ten facial attributes. These attributes are a subset the CelebA labels. All attributes and respective examples of the corresponding manipulation using SEGA depicted in Fig. S In addition to additive image edits, evaluated negative guidance of these attributes as well as two combinations of four simultancous edits, shown in Fig. 4 Therefore, empirical evaluation spans 21 attribute changes and combinations in total with all images being generated using the Stable Diffusion implementation.
We evaluated the generated images with a user study and provide more details on its implementation App. E The results are shown Tab. [ For positive guidance, SEGA faithfully adds the target concept the image average in 95% of the cases, with the majority of attributes exceeding 97%. We further manually investigated the two outliers, 'Bald' and 'Bungs'. We assume that many of the non-native assumption is based on correspondence with some of the workers, and the conspicuously low rate of annotator consensus. Consequently, the numbers for 'bangs' should be taken with grain of salt. For baldness, we found that long hair often makes up a large portion of portrait and thus requires more substantial changes to the image. Consequently, such edits require stronger hyperparameters than those chosen for this study. Furthermore, we observe negative guidance to remove existing attributes from an image work similarly well. It is worth pointing out that the guidance away from 'beard', 'bald' or gruy hair' usually resulted a substantial reduction of the respective feature, but failed remove entirely for 10% of the images. Again, this suggests that the hyperparameters were probably not strong enough.
annotators
not familiar with the term 'bangs' itself. This
Lastly, we looked into the simultancous guidance of multiple concepts once. The results in Tab. 2 empirically demonstrate the isolation of semantic guidance vectors. The per-attribute success rate remains similar for four instead of one distinct edit concept, suggesting interference of guidance vectors. Consequently, the success of multiple edits only depends on the joint probability of the individual concepts. In comparison, if only two out of four applied concepts were interfering with each other to be mutually exclusive, the success rate of such a combination would always be 0%. Contrary to that, successfully apply concurrent concepts in up to 91% % of generated images.
Additionally, we investigated any potential influence manipulations with SEGA on overall image quality. Utilizing the facial images from the previous experiment, we calculated FID scores against
7

---

Table I: Empirieal results of user study conducted on face attribules. Sample sizes result from portion of the 250 original images that did not contain the target attribute. Annotator consensus refers to the percentage of images for which the majority of annotators agreed on label. Success rate reported on those images with annotator consensus.
Table Results of user study simultaneous combination of face attributes. Sample sizes result from the portion of the 250 original images that did not contain sensus refers to the percentage of images for which annotator's i majority agreed on a label. Success rates for combinations on any combination attributes with per-attribute scores are reflecting the isolated success of that edit. Nonetheless, all scores shown for images with all edit con-
target attribute. Annotator con-
Attribute
#
Consensus( %
Success( %)
Gender
241
99.2
100.0
Glasses
243
99.6
100.0
cepts applied simultaneously.
Smile
146
100.0
99.3
Pos. Guidance
Bald
220
91.2
82.1
Attribute # Consensus( %) Success( %)
Beard
135
97.8
97.0
Attr.
100.0
Hat
210
99.0
99.0
2 Attr.
100.0
Combination
55
98.2
Curls
173
95.4
97.0
2 3 Attr.
98.2
Makeup
197
99.5
99.0
All 4 Attr.
90.7
Gray hair
165
97.6
91.2
Glasses
96.4
100.0
Bangs
192
86.2
82.7
Smile
100.0
96.4
55
Overall
1922
96.5
95.0
Curls
100.0
98.2
No Glasses
6
98.9
100.0
Beard
100.0
100.0
No Smile
93
100.0
94.4
1 Attr.
100.0
Neg. Guidance
No Bald
18
94.8
83.3
Combination 2
Attr.
45
81.8
100.0
Beard
100.0
89.9
2 Attr.
100.0
No Hat
31
100.0
90.3
All 4 Attr:
75.6
No Curls
50
98.0
98.0
No Smile
100.0
97.8
No Makeup
21
100.0
95.2
Makeup
100.0
97.6
45
Gray Hair
52
94.6
82.7
Hat
100.0
88.6
No Bangs
38
95.0
98.5
Female
100.0
100.0
Overall
420
91.9
92.1
reference dataset of FFHQ [1] FID scores for the original, unedited images are remarkable bad at 117.73, which can be attributed to small artifacts often present in facial images generated by Stable Diffusion. Editing images with SEGA significantly improved their quality resulting in an FID score on FFHQ of 59.86. Upon further investigation, we observed that the additional guidance signal for dedicated portion of the faces frequently removed uncanny artifacts, resulting in overall better quality,
Further, empirically evaluated SEGA the significantly different task of counteracting inap- propriate degeneration 3 29 For latent DMs, Schramowski et al. [29] demonstrated that an additional guidance term may be used mitigate the generation of inappropriate images. We demon- strate that SEGA'S 's flexibility enables to now perform this task on any diffusion architecture, both latent and pixel-based. To that end, we evaluated Stable Diffusion v1.5 and DeepFloyd-IF on the inappropriate- seeee-imaaepperre prompts (I2P) benchmark [29]. Subsequently, we suppressed inappropriate content using SEGA guide the generation away from the inappropriate concepts as specified by Safe Latent Diffusion (SLD) 29)]. We empirically chose the SEGA parameters to produce results similar the 'Strong' configuration of SLD. The results in Tab. ( demonsttatee that SEGA performs strong mitigation at inference for both architectures further highlighting the capabilities and versatility of the approach.
Comparisons. In addition to the isolated evaluation of SEGA'S 's capabilities, we conducted randomized user studies directly comparing SEGA with related methods. This comparison investigates the performance Composable Diffusion [16], Promp2Prompt [8). Disentanglement [35], and SEGA on tasks from four types of manipulation categories, reflecting broad range of operations. Specifically, we considered 1) composition of multiple edits, 2) minor changes, 3) style transfer and 4) removal of specific objects from scene. For each method and task, we selected the best-performing hyperparameters on a set of seeds 100 and subsequently generated fixed test (i.e., first applicable seeds >= 100). As first evaluation step, we conducted a user study to assess the success of an edit based on the presence/absence of the target attribute(s) shown in Tab, The comparatively low scores on removal tasks result from the methods often reducing presence of the targeted objects but not eliminating all of them in some cases. However, this peculiarity of the evaluation affects all methods similarly and does not influence the comparability capabilities. The results demonstrate that SEGA clearly outperforms Prompt2Prompt and Disentanglement on all
cherry-picked edits

---

Table 3: Results of Stable Diffussin and IF on the I2P benchmark 29
Values reflect the probability of generating inappropriate content (the lower the better) with respect to the joint Q16/NudeNet classifier proposed by Safe Latent Diffusion 29]. Subscript values denote the expected maximum inappropriateness over 25 prompts (the lower the better). For both architectures SEGA performs strong mitigation at inference.
Illegal
Harassment
Hate
Self-Harm
Sexual
Shocking
Violence
Overall
Activity
SD
0.32p.92
0.39g.96
0.350.94
0.40o.97
0.290.s6
0,511.00
0.42p.98
0.38g
SD w/ SEGA
0.12p.ea
0.150.76
0.09p.s7
0.07o.5a
0.040.36
0.190.84
0.14p.76
0.1lg.6s
IF w/ SEGA
0.350.9e 98
0.461.00
0.400, 99
0.40
0. 0.220.91
0.491.00
0.431.00
0.38g .9
0. 06
0.200.88
0.150.84
0.130.81
0.06o.58
0.21g.92
0.180.89
0.15g.84
Table 4: User study results of success rates for different manipulation types and methods. Overall SEga outperforms all compared techniques and specifically improves on tasks with multiple edits and small changes. Success rate is reported on those images where the majority of annotator agree on label.
Multi-
Minor
Style
Concept
Method
Conditioning (%)
Changes (%)
Transfer (%)
Removal (%)
Overall (%)
Composable Diffusion [6]
35.00
72.00e
100.0e
35.00
60.50.
Prompt2Prompt
35.00
68.00
65.00
5.00
43.25
Disentanglement 35
35.00
65.38
65.00
0.00
41.35
SEGA (Ours)
80.00
90.91
90.00o
30.00o
72.72
examined editing tasks. Compared to Composable Diffusion, SEGA again has significantly higher success rates for multi-conditioning and minor changes while achieving comparable performance for style transfer and object removal.
Beyond the simple success rate of edits, we evaluated the faithfulness to the original image com- position. To that end, users considered pair-wise comparisons of samples that both methods edited successfully and assessed the similarity with the original image. Importantly, users strongly prefer SEGA results over Composable Diffusion for 83.33% (vs. 13.33%) of samples. These two studies highlight that SEGA is generally preferred in terms of its edit capabilities and perceived fidelity over other methods. We present examples and further details on comparisons to related diffusion and StyleGAN techniques in App. G
In addition to the empirical evaluation, we present qualitative examples on other domains and tasks. We show further examples in higher resolution in the Appendix. Overall, this highlights the versatility of SEGA since it allows interaction with any of the abundant number of concepts diffusion models are capable of generating in the first place. We performed a diverse set of style transfers, as shown in Fig. 6 SEGA faithfully applies the styles of famous artists, as well as artistic epochs and drawing techniques. In this case, the entirety of the image has to be changed while keeping the image composition the same. Consequently, we observed that alterations to the entire output-as in style transfer-requee
Qualitative Results.
slightly lower threshold of X R: 0.9. Nonetheless, this still means that 10% of the c-space is sufficient to change the entire style of an image. Fig. ( also includes a comparison between outputs produced by SEGA with those from simple extensions to the prompt text. Changing the prompt also significantly alters the image composition. These results further highlight the advantages of semantic control, which allows versatile and yet robust changes.
6 Broader Impact on Society
21 28 have the potential for a far-reaching impact on society, both positive and negative, when deployed in applications such as image generation, image editing, or search engines. Previous research [3] 29 described many potential negative socictal implications that may arise due the careless use of such large-scale generative models. Many of these problems can be attributed to the noisy, large-scale datasets these models rely on. Since recent toxt-to-image models, such as SD, are trained on web-crawled data containing inappropriate content (30), they are no exception to this issue. Specifically, current versions of SD show signs of inappropriate degeneration 229 While Schramowski et al. (29 utilize the model's notion of inappropriateness to steer the model away from generating related content, it is noteworthy that we introduce an approach that could also be used to guide image generation toward inappropriate material. The limitations of Stable Diffusion also effect our editing of perceived gender which may
Recent developments in text-to-image models 25

---

Pevil y
Samala
Original ISD)
Original (S0)
Origiaal (Paela)
Origial (P)
Original
vxr
Drowing
Parelllarr
B/W Photo
SRGA
Promot
Figure 6: Qualitative examples using SEGA. Top row demonstrates that the approach is architecture agnostic showing examples for for Stable Diffusion Paella (26 and Deepfloyd-II Below we show style transfer using SEGA. All images are generated from the same noise latent using the text prompt 'a house at lake'. SEGA easily applies the characteristics of specific artist, epoch drawing/imaging technique the original image while preserving overall composition. In contrast, appending prompt with a style instruction results in images that significantly change the composition. (Best viewed in color)
exhibit potential biases. Since Stable Diffusion's ' learned representation of gender limited, restrict ourselves to binary labels, although gender expression cannot be ascribed to two distinct categories. Furthermore, does Stable Diffusion contain severe biases its attribution of gender and correlated features that may further reinforce pre-existing stereotypes. Since editing methods in general, and SEGA particular, rely on these representations of the underlying model, they will inevitably inherit them for related editing operations. However, on the positive side, SEGA has the potential mitigate bias. As demonstrated by Nichol et [2). removing data from the training set has adverse effects, e.g., on a model's 's generalization ability. In contrast, SEGA works at inference promoting fairness in the outcome. Therefore, we advocate for further research in this direction.
Another frequently voiced point of criticism the notion that generative models like SD are replacing human artists and illustrators. Indeed, certain jobs in the creative industry already threatened by generative image models. Furthermore, the collection of training data for (commercial) applications is often ethically questionable, building on people's 's artwork without their explicit consent. Nonetheless, we believe SEGA to be promoting creating artwork in an interactive process that requires substantial amount of iterative human feedback.
7 Conclusions
We introduced semantic guidance (SEGA) for diffusion models. SEGA facilitates interaction with arbitrary concepts during image generation. The approach requires no additional training, no extensions the architecture, no external guidance, is calculated during the existing generation process and compatible with any diffusion architecture. The concept vectors identified with SEGA robust, isolated, be combined arbitrarily, and scale monotonically. We evaluated SEGA variety of tasks and domains, highlighting-ande among soters--sphisticaaee image composition and editing capabilities.
Our findings are highly relevant to the debate on disentangling models' latent spaces. So far, disentanglement
property has been actively pursued [12]. However, it usually not necessary quality in itself but a means to an end to easily interact with semantic concepts. We demonstrated that this level of control is feasible without disentanglement and motivate research in this direction.
Additionally, we see several other exciting avenues for future work. For one, it is interesting to investigate further how concepts are represented in the latent space of DMs and how quantify them. Similarly, SEGA could extended to allow for even more complex manipulations, such as targeting different objects separately. More importantly, automatically detecting concepts could provide novel insights and toolsets to mitigate biases, as well as enacting privacy concerns of real people memorized by the model.
10

---

Acknowledgments This research has benefited from the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) cluster projects "The Third Wave of AI" and hessian from the German Center for Artificial Intelligence (DFKI) project "SAINT", the Federal Ministry of Education and Research (BMBF) project KISTRA (reference no. 13N15343), as well as from the joint ATHENE project of the HMWK and the BMBF "AVSV". Further, we thank Amy Zhu for her assistance in conducting user studies for this work.
References
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics, 2023.
[2] Yogesh Balaji, Seungjun Nah, Xum Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers. wrXiv222/.00322 2022.
[3] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan, Easily accessible text-to- image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAceT), 2022.
[4] Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), 2021.
[5] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the Advances Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[6] Kevin Frans, Lisa B. Soros, and Olaf Witkowski. Clipdraw: through language-image encoders. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeuriPS), 2022.
Exploring text-to-drawing synthesis
[7] Rinon Gal, Or Patashnik, Haggai Maron, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics, 2022.
[8] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prones image editing with cross attention control. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florint Balcan, and Hsuan-Tien Lin, editors, Proceedings of the Advances Neural Information Processing Systems: Annual Conference Neural Information Processing Systems (NeurIPS), 2020.
[10] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. urXiv:2207.12594 2022.
[11] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of thelEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[12] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[13] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), 2023.
[14] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho, Variational diffusion models. arXiv:2107.006300 2021.

---

[15] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In Proceedings of the International Conference on Learning Representations (ICL.), 2022.
[16] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In Proceedings of European Conference on Computer Vision (ECCV), 2022.
[17] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), December 2015.
[18] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR), 2013.
[19] Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the Ad- vances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2013.
[20] Tomás Mikolov, Wen-tau Yih, and Geoffrey Zweig, Linguistic regularities in continuous space word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACI-HILT), 2013.
[21] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022.
[22] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[23] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), 2019.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the International Conference on Machine Learning (ICML), 2021.
[25] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[26] Dominic Rampas, Pablo Pernias, Elea Zhong, and Mare Aubreville. Fast text-conditional discrete denoising on vector-quantized latent spaces. arXiv:2211.07292 2022.
[27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[28] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv:2205.1 /487, 2022.
12

---

[29] Patrick Schramowski, Manuel Brack, Björ Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[30]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-texl models. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[31] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Proceedings the International Conference on Machine Learning (ICML), 2015.
[32] Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference on Learning Representations (ICL.R, 2021.
[33] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on single image. ACM Trans. Graph., 2022.
[34] Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ACL, 2016.
[35] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to- imaie diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
13

---

A Implementation of SEGA
The implementation SEGA builds on the Stable Diffusion pipeline from the diffusers library As describe in Sec. 3 we calculate an additional noise estimate for each semantic concept. Consequently. we make one pass through the models U-Net for each concept description. Our implementation is included in the supplementary material, together with the necessary code and hyper-ypaaane parameters to generate examples shown the paper.
Additionally, we also provide the pseudo-code notation of SEGA in Alg Please note that this code slightly simplified to use single warm-up period é for all edit prompts ex-
B Numerical Properties of Noise Estimates
As discussed in Sec. 3.2 the numerical values of noise estimates are generally Gaussian distributed. This can be attributed to the fact that they are trained estimate a Gaussian sampled 6. We plotted exemplary numerical distributions of three noise estimates in Fig. 7 These estimates are an unconditioned 6g (z)),, text conditioned <g (z,; cp), and an edit conditioned ca (Zz ce) one. All
calculated at diffusion step 20. While these estimates lead to substantially different images, it is clear that their overall numerical distribution is almost identical.
Algorithm Semantic Guidance (SEGA)
Please note that this notation makes the simplifying assumption of one single warm-up period à for all edit prompts 6G-
condition textp, edit texts List(tearte) and diffusion steps T Ensure: 8m. E [0, 1], =i=o = 0, S. [0, 1), E (0, 1), st E [0, 20), 8y [0, 20), 5 [0. 20), 0
Require: model weights
DM + init- mait-diffusion-moddllle
Cp DM (
List(c.) DM. W.w.cncode(List(teens
latents +- DM.samplc(seed))
while * T do
reg 1hp <- DM pwM.reecictnooiselllennsss Cp)
List(n..) DM..prediic-noiieelaleetss List(c.))
g 8g * (mp- nh)
+
for all n in List(n.) do
di 4- n Tig
if negative guidance then of + -
end if
p4
+ where (+)) 2 No, s
< n4 sf
i+1
for
+ ΣiEt 9s * 7/
<- +
M44 3. v1 (1- 3m. 7/4
if 1 2
9 + + end
+ 9 latents DMuupdate-lteenn(prrdd,, latents) 4-- 141 end while
pred
14

---

text-canditioned estimate
edit-conditioned estimate
unconditioned estimate
0.3
Density
0.2
0.1
Numerical Value of Noise Estimate
Figure 7: Distribution of numerical values in different noise estimates. All estimates follow Gaussian distribution disregarding their different conditioning. Distribution plots calculated using kernel-density estimates with Gaussian smoothing. (Best viewed color)
Figure 8: Uniqueness of guidance vectors. The vector for 'glasses' is calculated once on the stre-d image and subsequently applied to other prompts (without colored border). On images where glasses added the guidance vector produces visual artifacts. (Best viewed in color)
C Further intuition and ablations hyper-parameters,
Let us provide some more detailed intuition for each of SEGA'S 's hyper-parameters
Scale Sc, The magnitude of the expression of a concept in the edited image scales monotonically with s.. Overall, the scale behaves very robustly for larger range of values. On the one hand, values of around 3 or 4 are sufficient for delicate changes. On the other hand, 8, is less susceptible to producing image artifacts for high values and can often be scaled values of 20+ w/o quality degradation if required for the desired expression of the concept.
Threshold X. SEGA automatically identifies the relevant regions of an image for each particular set of instructions and the original image in each diffusion step. Image regions outside these implicit masks (cf. Fig. 99) will not be altered by SEGA, leading to minimal changes. À corresponds to the portion of the image left untouched by SEGA. Consequently, higher values of À close to allow for minor alterations, whereas lower values may affect larger areas of the image. Depending on the edit, >= 0.95 will sufficient for most edits, such as small changes, whereas alterations of the entire image, like style transfer, usually require values between 0.8 and 0.9.
Warmup d In line with previous research, we have observed that the overall image composition is largely generated early diffusion steps, with later ones only refining smaller details. Consequently, the number of warmup steps may be used to steer the granularity of compositional changes. Higher values of 8 >= 15 will mostly change details and not composition (c.g., style transfer). Whereas strong compositional editing will require smaller values, i.c., 8 >= 5.
Momentum. Contrary to previous syper-parammters, momentum is more of an optional instrument to further refine the quality of the generated image. Most edits produce satisfactory results without the use of momentum, but image fidelity can further improve when used. Importantly, its main use
15

---

(a) Original image generated for prompt 'a photo of the face of a young man"
(b) Edited image using edit concept 'glasses'.
G
(c) Binary visualization of activating latent dimensions above SEGA threshold increasing diffusion steps. The dimensions of the latent edited by SEGA are closcly correlated with the region of the image in which the glasses are added.
Figure 9: Visualization latent spatial correlation SEGA thresholding.
case is in combination with warmup. Since momentum is already gathered during the warmup period, higher momentum facilitates higher warmup periods in combination with more radical changes.
We subsequently present visual ablations highlighting the effect of choosing different hyper parameter values. All images are generated from the same original image (shown in Fig. TT) obtained by the prompt 'a house at a lake'.
Figure 10: Unedited image generated from the prompt 'a house lake'. Included for reference of subsequent ablations.
C.1
Warmup-Scale
Visualization of the interplay between warmup 8 and semantic guidance scale s, shown in Fig.
C.2
Warmup-Threshold
Visualization the interplay between warmup and threshold À is shown in Fig. 12
C.3 Threshold-Scale
Visualization of the interplay between threshold and semantic guidance scale 8, is shown in Fig.
C.4 Momentum
Visualization of the interplay between the momentum scale 8m and momentum 3 is shown in Fig. 14
16

---

Scale Se 4
0
2
8
Warmup 8
124
16
Figure 11: Visualization of the interplay between warmup and semantic guidance scale Scx
Warmup
12
16
0.8
0.84
Threshhold
0.88
0.92
0.96
Figure 12: Visualization of the interplay hetween warmup and threshold
17

---

Scale Se
2
4
6
8
0.8
0.84
Threshold
0.88
0.92
0.96
Figure 13: Visualization of the interplay between threshold À and semantic guidance scale Se-
Momentum Scale Sm
0.0
0.2
0.4
0.6
0.8
0.0
0.2
Momentum Beta 3
0.4
0.6
0.8
Figure 14: Visualization of the interplay between the momentum scale and momentum 8.
18

---

crowd. crowded, people'
trees bush, bushes
trees, bush, bushes'
crowd, crowded, people'
Figure 15: SEGA offers strong control over the semantic space and can gradually perform edits at the desired strength. All images generated from the same initial noise latent using the prompt 'a crowded boulevard'. Editing prompts denoted in blue and are gradually increased in strength from left to right and top to bottom. (Best viewed color)
D Further Examples of Semantic Space's Properties
Here, we provide further examples of the properties of the semantic space created by SEGA (cf. Sec. 4). First, revisit the uniqueness of guidance vectors. As shown in Fig. 3b these vectors may be calculated once and subsequently applied to other prompts. However, this is limited to the same initial noise latent and requires similar image composition. We depict this difference in Fig. X where the guidance vector for 'glasses' is successfully applied to three images and fails to add glasses on three other images. However, the unsuccessfully edited images on the right retain the same image fidelity showcasing that SEGA'S 's guidance vectors do not introduce artifacts. Nonetheless, SEGA can easily incorporate glasses into these images by computing dedicated guidance vector (cf. Sec. 4p.
Next, we demonstrate that multiple concept vectors scale monotonically and independent of each other. Meaning that in nuuti-conditioning scenario the magnitude of each concept can be targeted independently. We depict an example in Fig. [5] Starting from the original image in the top left corner, remove monotonically reduces the number of people or trees until the respective concept is removed entirely, Again, the rest of the image remains fairly consistent, especially the concept that is not targeted. Going even further, a similar level of control is also possible with an arbitrary mixture of applied concepts. Let us consider the second row of Figure 15 for example. The number of trees is kept at medium level in that the row of trees on the left side of each image is always removed, but the larger tree the right remains. While keeping the number of trees stable, can still gradually remove the crowd at the same time.
concept from the image following the
y-axis. In both cases, SEGA
E
User Study
We subsequently provide further details the empirical evaluation presented in Sec. We attempted to label the generated images using a classifier trained on CelebA. However, despite the model achieving over 95% accuracy on the CelebA validation set, the accuracy for the synthetic images was
19

---

Examples of improved visual quality through SEGA editing. Uncanny artifacts in the original images (left) attenuated removed by the additional guidance in image regions edited using SEGA (right). The used edit 'smiling'.
Figure 16:
for some classes below 20%. Consequently, opted to rely human evaluators instead. For each of the 250 generated images we collected labels for all 10 attributes. Questions were posed in the form
Is the person wearing glasses?
or in the case of the gender at-
The three answer options were:
tribute:
Yes
Male
No
Female
Cannot Tell
Cannot Tell
Each user was tasked with labeling a batch 28 image/attribute pairs; 25 out of those were randomly sampled from our generated images and each batch contained hand-selected images from CelebA as sanity check. If users labeled the CelobA images incorrectly the batch was discarded and added back to the task pool. Each images/attribute combination was labeled by 3 different annotators resulting in annotator consensus at least 2 selected the same label.
Original
boat
monet'
'ooat' +' monet'
Figure 17: Robustness of SEGA across multiple tasks. Here style transfer and image composition are easily combined and executed simultaneously. The resulting image satisfies both tasks. All images generated using the prompt 'a castle next river', Top and bottom row use different seeds, (Best viewed in color)
20

---

To conduct our study we relied on Amazon Mechanical Turk where we set the following qualification requirements for our users: HIT Approval Rate over 95% and least 1000 HITs approved. Annotators were fairly compensated according to Amazon MTurr guidelines. Users were paid S0.70 for a batch of 28 images at an average of roughly minutes needed for the assignment.
In Fig. 16 we show examples of SEGA edits removing artifacts and improving overall incanny image feel. Consequently, the evaluated image fidelity of the edited images is higher than of the originally generated ones.
F
SEGA on various Architectures
As highlighted in the main body of this paper, SEGA can be integrated into any generative model using classifier-free guidance. Here we show further qualitative examples on various other architectures beyond Stable Diffusion.
implemented SEGA for two additional models: 1) Paella [26] which operates compressed and quantized latent space with iterative sampling over the distribution of
predicted tokens per latent dimension. 2) IF a publicly available pixel-Ievel diffusion model based on the Imagen architecture (8
G Comparisons to related methods
Subsequently, we compare SEGA to other methods for image editing.
G.1 Direct Comparisons
We first compare against related methods for diffusion that allow for direct comparisons, i.e. the same seed and prompt will generate the exact same original image. All of these techniques have dedicated failure modes, i.e. robustly and successfully performs the desired edits in all scenarios.
that these methods cannot accomplish. SEGA the other behaves more
G.1.1 Composable Diffusion
We first compare against Composable Diffusion [6] Fig 19 we can see that Composable Diffusion is also capable of conditioning on multiple concepts. However it makes more substantial changes than SEGA and produces artifacts when used with or more concepts. Additionally, CompDiff performs significantly worse image composition as shown in Fig. 20 For multiple of the applied
hutps://gihuub.co//eeeptob floyd/IF
original
'ast'
+ carroon emission'
"safety"
+ exppessive"
+'fast
carbon emission'
"safety"
Figure 18: Probing the diffusion model for learned concepts. Guiding towards or away from more complex concepts may reveal the underlying representation of the concept in the model. All images generated using the prompt 'a picture of car'. (Best viewed in color)
21

---

curly hair" glasses'
curly hair" + beardd' "glasses" I'smile
Original
gasses
glaass
SEGA
Composable Dillusion
Prompt2Prompt
Disentanglement
Figure 19: Comparison of multiple edits being performed simultaneously for various semantic diffuusion approaches.
Stazy Night
Origial
SEG4
Conponille Draaion
Figure 20: Comparison of SEGA against Composable Diffusion [16] image composition. SEGA signficantly outperforms Composable Diffusion terms of truthfulness the original image and combination of multiple concepts.
and combined concepts the original image no longer recognizable and SEGA also outperforms in terms of truthfulness of the targeted concepts and combinations thereof.
G.1.2 Prompt2Prompt
Prompt2Prompt 8 generally produces edits of similar quality as SEGA. However, Prompt2Prompt limited one editing prompt. This leads to the some concepts being ignored when aiming edit multiple ones simultancously as shown in Fig 19
G.1.3 Disentanglement
The Disentanglement proposed by Wu et al. 35 performs similarly well to SEGA in applying multiple edits faces T9). However, their method fails to realize small changes in an image as shown in Fig. 21 SEGA, on the other hand, faithfully executes the desired edit operation.
22

---

Original
crowd, people
Original
+jellly beans"
SEGA
Disentaglement
Figure 21: Comparison of SEGA against the approach of Wu et al. k55 for small edits. The Disentanglement
makes no significant changes to the image, whereas SEGA faithfully executes the desired edit operation.
trees, bushes crowd, people'
Original
crowd, people'
trees, bushes'
SEGA (Partial)
SEGA (Complete)
Negative Prompting
Figure 22: Comparison of SEGA against Negative Prompting. SEGA allows for dedicated control over the strength of the applied negative concept. Top row shows partial removal of the targeted concept with SEGA and middle row complete removal. Additionally, negative prompting breaks when used with multiple concepts simultancously.
Additionally, the Disentanglement approach requires the optimization of a linear transformation vector, whereas SEGA can be applied ad hoc.
G.1.4 Negative Prompting
SEGA offers multiple benefits over negative prompting as shown in Fig 22 Negative prompting offers no control over the strength of the negative guidance term, whereas SEGA provides a dedicated semantic guidance scale. 2) This often leads to unnecessarily large changes to the image when using negative prompts with the image oftentimes being completely different. With SEGA, however, changes can be kept to minimum. 3) Negative prompting does not allow for removal separate concepts, instead multiple concepts have to be provided in single prompt. This may lead to one of the concepts not being removed in the generated image. contrast, SEGA provides one guidance term with dedicated hyper parameters for each concept.
23

---

Original Recrotruction
glasses
+'curly hair
beard
SEGA
SLyPe-CLIP
Figure 23; Comparison of SEGA against StyleCLIP.
Original/
+'En
+ Edvard Munchi White Walker
Mona Uisa
Zombie
SEGA
SAyleGAN-NADA
Figure 24; Comparison SEGA against StyleGAN-VADA.
G.2
Related Approaches
We also compare against two StyleGAN based approaches for semantic editing. Notably, we had to reconstruct the original image with StyleGAN which results in differences in the un-edited image.
G.2.1 Style-CLIP
SEGA achieves edits of similar fidelity StyleCLIP 22 be applied ad hoc and does not require the time consuming mapping of prompt against a style space attribute as need for StyleCLIP, Furthermore SEGA works powerful diffusion model capable of generating diverse images beyond one domain, contrary to a dedicated StyleGAN for faces.
shown in Fig. 23 However, SEGA
G.2.2 StyleGAN-NADA
SEGA achieves edits of similar fidelity as StyleGAN-NADA as shown in Fig. 24 However, SEGA can be computed the fly and requires no tuning of the generative model as needed for StyleGAN-NADA. Furthermore SEGA works on powerful diffusion model capable of generating diverse images beyond one domain, contrary to dedicated StyleGAN for faces.
H Further Qualitative Evaluation
Lastly, provide further qualitative examples further highlighting the capabilities and use cases of SEGA.
The isolation of concept vectors (cf. Sec. 4 not only allows for multiple simultaneous edits of the same task but also arbitrary combinations. composition and change the style. The resulting image is faithful interpolation of both changes applied individually. This further highlights both the robustness of SEGA guidance vectors as well as their isolation.
Fig. 17 we simultaneously change the image

---

Griinal
Converible
Red Part
Liro
Oldime
SUV
Winter
the Doseet
the City
(a) Examples of image editing with prompt 'an image of car'. All images are generated from the same noise latent and text prompts. SEGA performs high quality local and global edits with minimal changes other aspects the image.
Original
Character Design
Original
Portroit Photo
Original
Animation
Concept Art
old
forest,
columns
(b) Optimization of artistic conception. Guidance conditioned on average embedding of prompl targeting one type of imagery. The resulting images high quality, reffect the targeted artistic style, while staying true the original composition.
Figure 25: Qualitative Examples of Semantic Guidance on various tasks and domains. (Best viewed in color)
In Fig. 25a we can see various edits being performed on an image car. This showcases the range of potential changes feasible with SEGA, which include color changes, altering the vehicle type or surrounding scenery. In each scenario, the guidance vector inferred using SEGA accurately targets the respective image region and performs changes faithful to the editing prompt. All while making next to no changes to the irrelevant image portions.
An additional benefit of semantic guidance directly on noise estimates is its independence on the modality of the concept description. the case of SD. the modality happens to be text in natural language, but SEGA can be applied to any conditional diffusion model. We explore this idea further with the goal of optimizing the overall artistic conception generated images. Instead of defining the target concept with one text prompt, we directly use an abstract embedding, representing particular type imagery. To that end, we collected prompts known to produce high-quality results for five different types of images in portrait photography, animation, concept art, character design, and modern architecture. The conditioning embedding for one style calculated as the average over the embeddings of all collected prompts. Exemplary outputs are depicted in Fig. 25b The results are of high quality and stay close to the original image accurately reflect the targeted artistic direction beyond single text prompt.
SEGA also contributes uncovering how the underlying DM "interprets" more complex concepts and gives further insight into learned representations. We perform this kind of probing of the model Fig. 18 For example, adding the concept 'carbon emissions' to the generated car produces seemingly much older vehicle with presumably larger carbon footprint. Similarly, reducing 'safery" yields a convertible with no roof and likely increased horsepower. Both these interpretations of the provided concepts with respect to cars are logically sound and provide valuable insights into the learned concepts of DMs. These experiments suggest a deeper natural language and image "understanding" that go beyond descriptive captions of images.
Prrmpps taken from http:://mpottta//hsss000oommmoo
25


Reviews:
[{"review_id": "Dwo2ZOVgKn", "reviewer": "Reviewer_2nZZ", "summary": "This paper analyzes classifier guidance in a more detailed way. The main idea is to tie the direction of guidance and the semantic, so that multiple guidance can be used. This idea is simple and effective. The paper also presents many small tricks to improve the performance. \n\nSEGA is evaluated on the face manipulation task and some qualitative benchmarks with a high successful rate.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Classifier-free guidance is a very interesting topic in diffusion models, which is the key to improve the performance but neglected by many theoretical papers.\n2. The paper tests their method on SD, Paella and DeepFloyd-IF, exhibiting its generality.", "weaknesses": "1. The paper presents a solid analysis, but not deep enough for classifier-free guidance. In my opinion, two things are very interesting:\n(1) What is the range of validity for the linearity of semantics? In word2vec, although the story of linearity is very appealing, but it is not true for many words. In diffusion model, there should be also a range to keep the linearity, which is very important for extending classifier-free guidance.\n(2) Figure 2 shows only the top 1-5% values are enough to guide the generation. This is an important finding, but it seems like a total statistic from multiple images. In a specific images, where are the top values distributed spatially? What is the relation between these values and initial noise?\n2. $\\mu$ is used to select the dims with the largest absolute values, which is the largest different between SEGA and usual CFG. However, more discussions about the motivation and quantitative results are needed. The importance of other tricks, e.g. warmup and momentum, are also unknown.\n3. The experiments are mainly limited to face-related and qualitative cases. I still think it could be better to compare SEGA with other baselines, e.g. prompt2prompt.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and will address each weakness individually. \n\n### W1: Analysis of classifier-free guidance\n1. **Linearity of Semantics.**\nWe would like to clarify that SEGA concepts scale monotonically and not necessarily linearly, as the space of noise estimates and the respective concepts in image space do not have a well-defined distance metric. Consequently, it is difficult to directly compare word2vec, for example. \nWe agree with the reviewer that this question is important for a deeper understanding of classifier-free guidance. In general, scaling reaches its limit when the numerical values are pushed out of their well-defined bounds, leading to image artifacts. We observed SEGA to be less susceptible to this issue than CFG, producing less artifacts at high guidance scales. We added a more detailed discussion on the limits of scaling to the paper. \n\n2. **Spatial distribution of SEGA dimensions**\nThe spatial distributions of activating SEGA dimensions above the threshold are closely correlated to the edited image region. We provide an example in the general response PDF above (Fig. 24). In the example of adding glasses to an image, these dimensions are concentrated on the location of the glasses in the image. Consequently, the spatial distribution changes based on the currently generated image and the semantically grounded location of the targeted edit. We believe these findings to be insightful in understanding classifier-free guidance and adjusted the paper to provide further clarification. \n\n### W2: Motivation of μ and importance of hyperparameters\nThe largest absolute difference is computed between the noise estimate of the edit concept and the unconditioned concept. In diffusion literature, this can be considered as the gradient of an implicit classifier steering the generation toward the edit concept. Consequently, the dimensions with the largest absolute value corresponds to the highest information density from this implicit classifier. As discussed above, these dimensions produce an implicit mask of the area to be edited while remaining semantically grounded in the currently generated image. \nWe included a more thorough discussion of this motivation in the paper. \nFurthermore, we added a more detailed discussion of the importance of each hyperparameter – including warmup and momentum – to the paper as outlined in the general comment above. \n\n### W3: Comparisons to other baselines \nWe conducted a user preference study comparing SEGA to Prompt2Prompt, Composable Diffusion, and Disentanglement. In summary, the results demonstrate that SEGA generally outperforms all these methods with more details being provided in the general response above.\n"}, {"review_id": "QjlicBRQgX", "reviewer": "Reviewer_i1VN", "summary": "This work proposes SEGA (semantic guidance) for text-to-image diffusion models. It is a architecture-agnostic technique and is only applied during the sampling process. In addition to classifier-free guidance, a semantic guidance is applied that pushes noise prediction along the direction of a specific semantic concept.", "questions": "Choices of $c_e$ presented in the paper are all single-word, precise semantic concepts. You tried combining multiple concepts via Eq.10. Have you thought about making $c_e$ a composite concept?", "limitations": "The authors have discussed the limitations and broader impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The technique is simple and intuitive. It achieves fine-grained semantic control of image generation via noise-estimate manipulation, therefore an easier process than image-editing models. The results look interesting.\n", "weaknesses": "The human study only evaluates on whether a label is present/absent. No study is done on the quality of images. For example, adding 'glasses' on an image hopefully will not alter the other attributes of the image, i.e. the person still looks the same, the facial expression remains the same, etc. And in addition, the image quality also should not drop significantly. These can be done via user study, or image quality can be measured by inception scores or FID. (I think at least some measure of image quality comparison with the unaltered images is needed.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer i1VN for their input and questions. Below we address each weakness and question individually. \n\n### W1: Image quality measurement. \nWe agree with the reviewer that a measure of image quality comparison with unaltered images will improve the paper substantially and conducted a respective experiment. In summary, these additional results give no indication of quality degradation due to SEGA editing. In fact, the scores improve significantly for the edited images.\n\nAs suggested, we calculated FID scores of generated facial images against FFHQ for the original images and edited ones generated for the experiments in Tab 1 & Tab 2. \nThe FID score for the original images are considerably low at 117.73, which can be attributed to small artifacts often present in facial images generated by Stable Diffusion. \nEditing images with SEGA significantly improved their quality resulting in an FID score on FFHQ of 59.86. Upon further investigation, we observed that the additional guidance signal for a dedicated portion of the faces frequently removed uncanny artifacts, resulting in overall better quality. We include exemplary images highlighting this behavior in the general response PDF (Fig. 25).\nAdditionally, we conducted a preference selection user study, including comparisons of edit quality. Further details are provided in a general comment above. \n\n### Q1) Composite concepts \nSEGA may also be used with a composite concept prompt. In fact, the experiment conducted on suppressing the generation of inappropriate content (Tab. 3) uses the composite concept defined by SLD to describe unwanted material: \n‘hate, harassment, violence, suffering, humiliation, harm, suicide, sexual, nudity, bodily fluids, blood, obscene gestures, illegal activity, drug use, theft, vandalism, weapons, child abuse, brutality, cruelty’\nAs demonstrated by the classification results over different categories, SEGA actively suppresses inappropriate material for each of these categories with only one edit prompt containing multiple concepts.\n"}, {"review_id": "vajXP5y14f", "reviewer": "Reviewer_q8hy", "summary": "The paper proses a novel approach called Semantic Guidance (SEGA) to edit an image using text only and without fine-tuning the T2I model. It does this by manipulating the noise estimate in the T2I diffusion model. The space of sparse noise-estimate vectors is referred to as the semantic space. The paper justifies various desirable properties of SEGA such as altering fine-grained output details, architecture agnostic (as long as it uses classifier-free guidance), works on both pixel and latent space, no T2I fine-tuning, supports editing of multiple attributes (addition or deletion). Guidance vector is unique per concept and thus can be pre-computed. The value of semantic guidance vector controls the magnitude of semantic concept (e.g. degree of smile). Different concepts/modifiers typically affect different regions of the noise estimate.  Experiments in the paper has good qualitative results as well as positive human evaluation on  2 data sets.", "questions": "Please see list of weakness", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1) Clarity\n\nS2) Simplicity, elegance, effectiveness, generality of the proposed approach\n\nS3) Several useful insights regarding the semantic vector\n\nS4) Good qualitative results and human evaluation\n", "weaknesses": "W1) Quantitative evaluation on a large data set. E.g. face attribute detection could be used to do large scale evaluation\n\nW2) Not sure how effective SEGA is for more complex manipulations involving multiple objects in the same image e.g. controlling the color of 2 objects independently using referring expressions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their great feedback and address each weakness separately. \n### W1: Large-scale evaluation w/ face attribute detection\nIn line with the reviewer’s suggestion, we initially attempted to use face attribute detection for large-scale evaluation. As outlined in the Appendix, however, the drastic shift between labeled, real images and Stable Diffusion generated synthetic images made an automatic evaluation infeasible. Presumably, also due to the oftentimes generated uncanny artifacts in facial images (cf. Fig.25  in general response PDF). In fact, we trained a classifier on CelebA, achieving over 95% on a held-out validation set that we attempted to use for labeling images edited with SEGA. However, an investigation of this classifier’s performance for synthetic images resulted in less than 20% accuracy for glasses, for example, rendering such a method impractical for the evaluation of SEGA. \n\n### W2: Complex manipulations\nComplex manipulations targeting different objects independently are indeed possible with SEGA. As shown in the general response PDF, SEGA’s threshold λ identifies implicit masks of regions in an image corresponding to the desired edit. Consequently, manipulating two concepts independently requires a set of edit expressions and thresholds that separates the objects accordingly. We agree with the reviewer that this is a promising research direction and are currently investigating such complex manipulations for future work. We extended the discussion in the conclusion accordingly.\n"}, {"review_id": "JMyphfg1En", "reviewer": "Reviewer_ZrN7", "summary": "This paper proposes a method for attribute editing of the images generated with Text-to-Image Diffusion models (T2I-DM). This is done in a manner similar to CFG, where extra edit prompts are passed alongside the text prompt to generate the image. Noise predicted from the edit prompt in the denoising process guides the editing of the generated image. The properties of SEGA are extensively reported with qualitative results and on multiple T2I-DMs.", "questions": "Insights about the hyperparameter choices in the supplementary could be presented better. The conclusion statements in each case are missing. What aspect of the generation (image quality, details etc) are controlled by each of the hyperparameter? Perhaps a summary of the hyperparameter analysis could be made part of the main paper, instead of Fig. 6 and its analysis. ", "limitations": "Limitations have been addressed adequately.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The key idea is simple, well-motivated, and theoretically backed.\n2. The four semantic properties of the method are extensively studied and experimentally backed.\n3. Human evaluation results prove the effectiveness of the method.\n", "weaknesses": "1. In Table 1, I suppose the 250 images could or could not contain an attribute (for eg. smile). In that case, the results have been reported by adding smile to 146 images which did not contain it and removing smile from the 93 images that previously had a smiling face. If so, the same protocol could be followed for all other attributes. However, the results for negative attributes are reported only for 3/9 attributes (excluding gender).\n\n2. It would complete the paper if a user preference study against competitive methods such as Wu et al. [1] are also reported.\n\n[1] Wu et al. Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models. https://arxiv.org/abs/2212.08698\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ZrN7 for their feedback and subsequently address each stated weakness and question. \n\n### W1: Additional results for negative attributes \nWhile we believe the presented attributes for negative guidance to be representative of SEGA’s capabilities, we agree that the results on negative guidance for the additional attributes would provide a more holistic view. Consequently, we performed the respective experiments with the following results: \n\n|   Attribute   |   Samples   |   Consensus (%)   |   Success (%)   |\n| --- | --- | --- | --- |\n|   No gray hair   |   52   |   94.55   |   82.69   |\n|   No curly hair   |   50   |   98.04   |   98.00   |\n|   No bald   |   18   |   94.74   |   83.33   |\n|   No makeup   |   21   |   100.0   |   95.24   |\n|   No hat   |   31   |   100.0   |   90.32   |\n|   No bangs   |   38   |   95.00   |   89.47   |\n|   **Total** |  **210**   | **96.77** |  **90.00**  |\n\n\nThe overall success rates remains very similar to the results reported in the paper, thus providing further evidence of the capabilities of SEGA.\n\n### W2: Comparative user study\nWe conducted a user preference study comparing SEGA to Prompt2Prompt, Composable Diffusion, and Disentanglement. In summary, the results demonstrate that SEGA generally outperforms all these methods, with more details being provided in the general response above.\n\n### Q1: Hyperparameter analysis \nWe provide further details on each hyperparameter in a general comment above. Given the additional page provided for the camera-ready version, we would include this analysis in the main body of the paper with no need for removing other content. \n"}, {"review_id": "e1MTORriDX", "reviewer": "Reviewer_Lm7m", "summary": "The paper introduces Semantic Guidance, a method that enables user control and interaction with text-to-image diffusion models to generate images that align with their intended semantics. The authors highlight the challenge that small changes in the input prompt can result in vastly different images. To address this issue, SEGA leverages the previously proposed composable diffusion approach to allow for fine-grained control over the diffusion process. The experiments showcase the effectiveness of SEGA mostly on human faces. ", "questions": "1. What are the typical failure cases of the proposed method? ", "limitations": "The limitation of the method and evaluation setting needs to be discussed. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies an important problem, fine-grained control for diffusion models. \n\n2. Experiments on human faces are extensive. Both automatic evaluation and human evaluation are conducted. \n\n3. Results show that SEGA enables subtle edits on human faces, which disentangles the intended edits from factors that should remain untouched. \n", "weaknesses": "1. **Similarity to previous work:** The method is almost identical to Composable Diffusion [1], where different concepts are composed to manipulate the generated image. \n\n2. **Limited scope of evaluation:** Most experiments are done on human faces. It is unclear if the method can be generalized to other object categories, e.g., the height of a building, the width of a bench, or moving an object left/right. \n\n3. **Potential biases:** Several semantic edits target controversial concepts such as gender, hate, and violence. It's better to discuss what is the scope of evaluation here, e.g., what genders are considered and what remains unresolved. \n\n[1] Compositional Visual Generation with Composable Diffusion Models. 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and suggestions. We will address each of the stated weaknesses and questions individually. \n\n### W1: Similarity to previous work\nWe respectfully disagree with the reviewer that our work is almost identical to Composable Diffusion. We agree that there are some similarities between the approaches, which is why we discuss Composable Diffusion in our related work.\nAs highlighted by the other reviewers, we argue that SEGA makes a relevant contribution by identifying concept encoding dimensions of the noise estimates, as well as other manipulation techniques yielding fine-grained semantic control. \nThe improvements provided by SEGA are also clearly highlighted in the comparison user study we provide in the general comment above. When compared to ComposableDiffusion SEGA offers better success rates and editing quality for tasks related to multi-conditioning and small changes while providing preferred edits in cases of style transfer and removal at comparable success rates. \n\n### W2: Limited scope of evaluation\nWe would like to point out that the paper does provide an extensive, empirical study beyond human faces, as presented in Tab. 3. Additionally, we believe that the user study presented in the general comment above, which compares SEGAs performance to other techniques on a general set of editing tasks, addresses the limitation in evaluation stated by the reviewer. \n\n\n### W3: Potential biases\nWe agree with the reviewer that the paper would benefit from a more nuanced discussion of ethical and societal implications. Consequently, we adjusted the paper to better reflect the following biases and limitations: \n1. We acknowledge that the editing of perceived gender in the paper may exhibit potential biases due to the following factors. Due to the limitation of Stable Diffusion’s learned representation of gender, we restrict ourselves to binary labels, although gender expression cannot be ascribed to two distinct categories. Furthermore, does Stable Diffusion contain severe biases in its attribution of gender and correlated features that may further reinforce pre-existing stereotypes. Since editing methods, in general, and SEGA in particular, rely on these representations of the underlying model, they will inevitably inherit them for related editing operations.\n2. The results presented on the suppression of inappropriate content are based on the definitions of inappropriateness introduced by the authors of Safe Latent Diffusion. We agree with the authors that what is considered inappropriate imagery may differ based on context, setting, cultural and social predisposition, and individual factors and is highly subjective overall. Again using the representations of the model itself assumes a definition of inappropriateness that may appear biased to some\n3. While we believe SEGA to be an amazing tool promoting creating artwork in an interactive process, we want to acknowledge the group of artists feeling threatened by the wide accessibility of these models. Additionally, we want to highlight the ethical implications of using web-scraped data for the training of generative models. Data that contains the artwork of people who have not provided explicit consent for the usage in ML training. \n\n### Q1: Typical failure cases \nAs with all editing methods, a key failure case is attempting edits for concepts the underlying model has no clear representations of. While these cases are limited in a generalistic model such as stable diffusion, examples can include unknown artists or highly specific domains such as medical imaging. \nAdditionally, targeting only one of several same objects (e.g., two people) is challenging, but also not impossible. We believe that further improvements to the masking process (c.f. Fig. 24 in the general response PDF) can be a viable solution for this issue which we are currently investigating for future work."}]

Author rebuttal:
We thank all reviewers for their valuable feedback and suggestions, which we have taken under careful consideration to improve the paper. 
## Comparison 
As suggested by the reviewers, we conducted randomized user studies comparing the capabilities of SEGA with related methods. Namely: Prompt2Prompt (P2P) [8], Disentanglement [35], and Composable Diffusion (CompDiff) [16]. 
To that extent, we considered tasks from four types of manipulation categories, reflecting a broad range of operations. 
1) Composition of multiple edits similar to the experiment reported in Tab. 1 & 2. 
2) Minor changes (e.g., change of pizza toppings, turning a car into a convertible, etc. )
3) Various style transfers
4) Removing a set of specific objects from a scene (e.g., trees from a view of a landscape)

For each method, we selected the best-performing hyperparameters on a set of seeds < 100 and subsequently generated non-cherry-picked edits on a fixed test set (i.e., first applicable seeds >= 100). 
We subsequently asked users to assess the success of an edit based on the presence/absence of the target attribute(s). Success rates per category are as follows: 

|Method|Multi-Condition (%)|Minor Changes (%)|Style Transfer (%)|Remove (%)
|--|--|--|--|-- 
|SEGA|80|90|90|30
|Composable Diffusion|35|72|100|35
|Prompt2Prompt|35|68|65|5|
|Disentanglement|35|65|65|0|

We would like to note that the comparatively low scores on removal tasks result from the methods often reducing the presence of the targeted objects but not eliminating all of them in some cases. However, this peculiarity of the evaluation affects all methods similarly and does not influence the comparability of method capabilities. 

These results demonstrate that SEGA clearly outperforms P2P and Disentanglement on all examined editing tasks. Compared to CompDiff, SEGA again has significantly higher success rates for Multi-Conditioning and Minor changes while achieving comparable performance for style transfer and object removal. 

Lastly, we investigated user preference with respect to the degree of changes made by each method. To that end, we looked at pair-wise comparisons of samples that both methods edited successfully and asked users to asses the similarity with the original image. 

Compared to Disentanglement, users strongly prefer SEGA results for 83.33% (vs. 13.33%) of samples, indicating the generally better performance of SEGA in terms of success rate and edit preference. 

When compared to CompDiff users also strongly prefer the SEGA-edited images (80% vs 15% overall). Consequently, SEGA offers better success rates and editing quality for tasks related to multi-conditioning and small changes while providing preferred edits in cases of style transfer and removal at comparable success rates (83.33% vs. 8.33% preference for style and 100% vs. 0% preference for removal). 

Lastly, comparing SEGA to P2P, our method offers better success rates across all categories. However, users preferred the edits of P2p for minor changes and style transfer in cases where both methods performed successful edits (17.86% vs. 82.14%). Nonetheless, SEGA remains more versatile in general while also providing methodological benefits, such as the fact that SEGA edit instructions can be provided in an independent manner without dedicated connections to certain tokens of the original prompt required for P2P. Additionally, we argue that the granularity of edits on a single sample can easily be adjusted to the user’s preferences using the sophisticated hyperparameters provided by SEGA.

[8] P2P arXiv:2208.01626

[16] CompDiff arXiv:2206.01714

[35] Disentanglement arxiv:2212.08698

## Hyperparameter analysis
As suggested by the reviewers, we added a more detailed discussion on the influence of each individual hyper-parameter, which can be summarized as follows:

**Scale s_e**. The magnitude of the expression of a concept in the edited image scales monotonically with s_e. Overall, the SEGA scale behaves very robustly for a larger range of values. On the one hand, values of around 3 or 4 are sufficient for delicate changes. On the other hand, s_e is less susceptible to producing image artifacts for high values and can often be scaled to values of 20+ w/o quality degradation if required for the desired expression of the concept.

**Threshold λ**. SEGA automatically identifies the relevant regions of an image for each particular set of instructions and the original image in each diffusion step. Image regions outside these implicit masks (cf. Rebuttal PDF Fig. 1) will not be altered by SEGA, leading to minimal changes. λ corresponds to the portion of the image left untouched by SEGA. Consequently, higher values of λ close to 1 allow for minor alterations, whereas lower values may affect larger areas of the image. Depending on the edit, λ >= 0.95 will be sufficient for most edits, such as small changes, whereas alterations of the entire image, like style transfer, usually require values between 0.8 and 0.9. 

**Warmup δ**. In line with previous research, we have observed that the overall image composition is largely generated in early diffusion steps, with later ones only refining smaller details (cf. Rebuttal PDF Fig. 26). 
Consequently, the number of warmup steps may be used to steer the granularity of compositional changes. Higher values of δ >= 15 will mostly change details and not composition (e.g., style transfer). Whereas strong compositional editing will require smaller values, i.e., δ >= 5. 

**Momentum**. Contrary to previous hyper-parameters, momentum is more of an optional instrument to further refine the quality of the generated image. Most edits produce satisfactory results without the use of momentum, but image fidelity can further improve when used. Importantly, its main use case is in combination with warmup. Since momentum is already gathered during the warmup period, higher momentum facilitates higher warmup periods in combination with more radical changes. 

Meta-review: The AC has carefully read the paper, reviews, author response, and discussions. This paper proposes semantic guidance that pushes noise prediction along the direction of a specific semantic concept. The idea is simple and interesting. 3 out of 4 reviewers are positive. The AC agrees with the reviewers that the paper presented a simple yet effective approach for attributing editing in the diffusion process, which will be of wide interest to the ML community. And thus recommend acceptance. However, the authors are strongly encouraged to follow reviews to polish the paper, for example, adding more comparisons with related works.
Decision: Accept (poster)