Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions

## Abstract

Abstract In recent years, advances in text-to-image (T2I) diffusion models have substantially elevated the quality of their generated images. However, achieving fine-grained control over attributes remains a challenge due to the limitations of natural language prompts (such as no continuous set of intermediate descriptions existing between “person” and “old person”).
Even though many methods were introduced that augment the model or generation process to enable such control, methods that do not require a fixed reference image are limited to either enabling global fine-grained attribute expression control or coarse attribute expression control localized to specific subjects, not both simultaneously.
We show that there exist directions in the commonly used token-level CLIP text embeddings that enable fine-grained subject-specific control of high-level attributes in text-to-image models. Based on this observation, we introduce one efficient optimization-free and one robust optimization-based method to identify these directions for specific attributes from contrastive text prompts. We demonstrate that these directions can be used to augment the prompt text input with fine-grained control over attributes of specific subjects in a compositional manner (control over multiple attributes of a single subject) without having to adapt the diffusion model. Project page: compvis.github.io/attribute-control . Code is available at github.com/CompVis/attribute-control .

## 1 Introduction

Text-to-image (T2I) models have recently seen considerable progress in their capabilities and the quality of their generated images.
However, a persistent challenge lies in achieving fine-grained control over the generated images, particularly concerning attribute expression. This can be attributed to the limitations of natural language, which does not allow for the very fine-grained description of attribute expression in a way that diffusion models reliably understand.
Unlike image editing use cases, where a base image is given, and masks can be provided to adapt a global method to only affect a target subject, we target the pure generation use case, where no reference image is given. This means that subject instances can only be identified via the prompt.
Even though many methods (see [Tab. 1](#S2.T1)) were introduced that augment the model or generation process to enable such control without per-image optimization (in contrast to Imagic []), they are limited to either enabling global fine-grained attribute control [, , , , ] or coarse attribute control localized to specific subjects [, ], not both simultaneously.

Starting from a simple prompt for a T2I diffusion model, our goal is to influence the generation process in a fine-grained manner, in both localization and magnitude. More specifically, in a prompt such as “a photo of a woman standing next to a man in front of their car”, we want to influence attributes of each subject – “woman”, “man”, and “car” – separately and with fine-grained control over their individual attribute expression.
Typically, one would add these attributes to the subjects in the prompt (“old person”), but this approach yields only very coarse control of these attributes. Additionally, when composing multiple of such attributes in the prompt, diffusion models often ignore a subset of them [, ].

We show that there exist directions in the commonly used token-level CLIP [] text embeddings that enable fine-grained subject-specific control of high-level attributes in T2I models, as opposed to inversion methods that learn instance information [, ]. Based on this observation, we introduce methods to identify these directions for specific attributes and show how they can be used to augment the prompt text input with fine-grained continuous control over attributes of specific subjects in a compositional (stackable) manner without adapting the diffusion model or incurring additional costs during generation.

We summarize our main contributions as follows:

- •
We show that token-level edit directions that allow fine-grained control of subject-specific attributes exist in common CLIP text embeddings and that diffusion models are capable of interpreting them
- •
We show that T2I diffusion models are capable of backpropagating high-level semantic concepts to their text embedding input as adaptations of existing embeddings using just the reconstruction loss objective on a single image
- •
We introduce two approaches for identifying those directions for specific attributes or concepts from contrasting text prompts describing these concepts, one simple optimization-free method and one optimization-based one that identifies more robust directions
- •
We show that these token-level edit directions enable fine-grained, subject-specific, compositional control of attributes and concepts in common diffusion models

## 2 Related Work

This section provides an overview of work related to our method. We also compare our method with others in [Tab. 1](#S2.T1).

#### 2.0.1 Influencing the Generation Process of Diffusion Models

Since Diffusion Models do not provide the same ordered and interpretable latent space as Generative Adversarial Networks [, , ] do, a considerable research interest has formed to provide Diffusion Models with greater ability to control visual detail.
A common approach for this is to directly use new or adapt existing neural features to guide the generation process towards semantic changes [, , , , ]. This allows for fine-grained and even non-text-based instructions to appropriately affect the sample, like interpolating between specific concepts [, , , ], subject-specific control [] or entirely erasing concepts []. Our work encompasses the interpolating abilities as well as subject-specific control in complex samples, therefore combining two fundamental properties that have been mutually exclusive for generic methods so far.

##### Image Editing.

One special case of influencing the generation process of diffusion models is the case where a reference image is given. Here, the expectation typically is that
only a few aspects of the image should be altered.
Numerous prior works [, , , , ] have shown difficulties properly disentangling multiple concepts in a sample to prevent global changes and enable a user to locally concentrate edits. To approach this problem, one can directly integrate specific changes into the image using the reverse diffusion process [], by inversion [, ] or by marking regions using edit masks [, ] to enforce localization.

#### 2.0.2 Instance Personalization

Another related task is personalizing diffusion models to enable them to generate images that contain specific instances of subjects.
Approaches based on finetuning include DreamBooth [], which leverages a prior preservation loss to adapt the visual backbone, while approaches like Textual Inversion [] restrict the adaptation of the model to an added embedding vector to be optimized to represent a specific instance or concept. Furthermore Kumari *et al*. [] propose CustomDiffusion for efficient training of multiple concepts by finetuning only cross-attention layers. Moreover, gaining enhanced control in scenarios where samples exhibit complex compositional structures remains a significant challenge for text-to-image models []. A prominent strategy to achieve controllable image synthesis involves the use of energy-based models, as suggested by several previous works [, , ].

#### 2.0.3 Global Edit Directions in CLIP Space

A wide range of previous works [, , , ] investigated the use of the CLIP embedding space [] to guide the image generation process of StyleGAN []. These studies identify directions within CLIP’s embedding space that correspond to global semantic changes and utilize these directions to steer the generation process.

Table: Table 1: High-level comparison of our method with other methods that allow attribute control in T2I diffusion models.

## 3 Method

T2I Diffusion models [, ] model a reverse diffusion process $p_{\theta}(\mathbf{x}_{0:T}|\mathbf{c})$ that enables sampling from the distribution of images $p_{\theta}(\mathbf{x}_{0}|\mathbf{c})$ given a text prompt condition $\mathbf{c}$ and a Gaussian noise sample $\mathbf{x}_{T}$. They iteratively denoise $\mathbf{x}_{T}$ using a diffusion model $\hat{\mathbf{x}}_{0,\theta}(\mathbf{x}_{t}|\mathbf{c},t)$ which (implicitly) predicts the clean sample given a noised image. This conditioning $\mathbf{c}$ is typically obtained using a CLIP [] text encoder $\mathcal{E}_{\mathrm{CLIP}}$ as a tokenwise embedding $\mathbf{e}=\mathcal{E}_{\mathrm{CLIP}}(\mathrm{prompt})$.

#### 3.0.1 Exerting Control over the Generation Process in Diffusion Models

We aim to influence the generated samples $\mathbf{x}_{0}\sim p_{\theta}(\mathbf{x}_{0}|\mathbf{e})$, more specifically the expression $\mathrm{expr}(A_{i})$ of specific attributes $A_{i}\in\mathcal{A}$ of a specific subject $S_{j}\in\mathcal{S}$. This subject is described in the prompt at $\mathrm{prompt}_{[S_{j}]}$ (with the corresponding embedding $\mathbf{e}_{[S_{j}]}$) and is located in the generated image in $\mathbf{x}_{0,[S_{j}]}$. The location of the subject in the generated image $\mathbf{x}_{0,[S_{j}]}$ is generally not known ahead of time, as it is dynamically determined during the generation process. Specifically, we aim for a change of the expression of $A_{i}$ of subject $S_{j}$.

To control the expression of an attribute $A_{i}$, the nature of this image generation approach enables a limited number of aspects to target for influencing the generation process.
To control the global expression $\mathrm{expr}_{\mathrm{global}}(A_{i})$, one option is to modify the diffusion model by re-defining the expression of attributes on a global level [, , ].
Another possibility is to directly modify the starting noise latent $\mathbf{x}_{T}$ [] or the intermediate noise latents $\mathbf{x}_{t}$, either using dynamically predicted [, , , , , ] or fixed [] auxiliary directions.
Alternatively, one can also influence how the diffusion model interprets the prompt embedding $\mathbf{e}$ []. This allows for instance specificity through the prompt but does not work well for continuous attribute expression control (see [Fig. 7](#S4.F7)c). Finally, one can directly modify the prompt embedding $\mathbf{e}$, influencing the text encoder $\mathcal{E}_{\mathrm{CLIP}}$, or modifying the original tokens. Methods like [, , , ] previously investigated this approach for inserting instance appearance information, but not for fine-grained attribute control.

As it already directly enables instance-specific coarse attribute control via text, the tokenwise prompt embedding $\mathbf{e}$ is a natural choice as the basis for a method for fine-grained attribute control, both in expression and instance-specificity. For this to be practical, multiple requirements have to be fulfilled:
i) The diffusion model has to be capable of interpreting modified prompt embeddings $\mathbf{e}^{\prime}$ that do not directly correspond to a possible text prompt.
ii) Fine-grained changes of the token-wise prompt embedding have to be localized in what they affect (specifically relating to a subject $S_{j}$ in both prompt embedding $\mathbf{e}$ and image $\mathbf{x}_{0}$). This localization has to be discoverable and interpretable.
iii) The tokenwise CLIP embedding space, as interpreted by the diffusion model, has to locally (around anchor points from real words) approximate a Euclidean manifold that behaves similarly across similar anchor points to enable composable smooth local edits along fixed category-specific directions $\Delta\mathbf{e}$.
iv) These directions $\Delta\mathbf{e}$ have to be practically discoverable.
We investigate requirements i-iii) in [Sec. 3.1](#S3.SS1) and iv) in [Sec. 3.2](#S3.SS2) and [Sec. 3.3](#S3.SS3).

### 3.1 Learning Semantic Edits from Text/Image Pairs

Let us now investigate whether the previously mentioned conditions i-iii) are fulfilled, and semantic directions that can be applied on top of embeddings of actual captions exist in the tokenwise CLIP embedding space.

A wide range of previous works [, , , ] found that the reconstruction loss of a pre-trained T2I diffusion model can be used to backpropagate instance appearance information to the prompt embedding.
These instance text embeddings can then enable various personalization and image editing use cases in the generation process.
This implies that the first condition – the diffusion model can interpret points that do not exactly lie in the text embedding space that the CLIP model provides – is given. Otherwise, simple gradient descent-based learning of very specific instance appearance information would not be plausible.

Although the objective is only based on pixel-wise reconstruction information, we find that this general approach can also learn semantic information directly. Using a single image/caption pair $(\mathbf{x}_{0},\mathrm{prompt})$, we apply a random amount of noise to the image and backpropagate the reconstruction loss

$$ $\mathcal{L}(\mathbf{x}_{0},\mathbf{e}+\Delta\mathbf{e})=\mathbb{E}_{\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I}),t\sim\mathcal{U}(0,T]}\!\left[w(t)\left\|\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}(\alpha_{t}\mathbf{x}_{0}+\sigma_{t}\boldsymbol{\epsilon}|\mathbf{e}+\Delta\mathbf{e},t)\right\|_{2}^{2}\right]$ (1) $$

through the diffusion model $\hat{\mathbf{x}}_{0}(\cdot)$. We update a learnable delta $\Delta\mathbf{e}$ that is added to the prompt embedding $\mathbf{e}$ to minimize the regularized(^2^22Implemented as weight decay through AdamW [] on $\Delta\mathbf{e}$.) reconstruction loss. One crucial detail is that the noise $\boldsymbol{\epsilon}$ is randomly re-drawn at every step.

We find that this approach, indeed, leads to learned prompt embedding deltas that capture semantic differences between the set of generated images given just the prompt and the target image. [Fig. 2](#S3.F2) shows that they substantially reduce the semantic gap between the image generated with the original prompt and the target image.
Additionally, linear interpolation between the original prompt embedding and the embedding with the delta applied shows a clear semantic progression from the originally generated image toward the target image. This indicates that the tokenwise CLIP embedding space, as interpreted by the diffusion model, is at least locally smooth in semantic meaning (condition iii). However, we also observe “phase changes” where the image changes substantially during a short subset of the interpolation trajectory for large changes, indicating that the embedding space is not globally smooth.

Figure: Figure 2: We observe that starting from a target image (rightmost column) and a corresponding caption (leftmost column), edits $\Delta\mathbf{e}$ to the tokenwise prompt embedding $\mathbf{e}$ can be learned. Starting from the unmodified $\mathbf{e}$, these edits go from the generated images (unmodified images are marked in orange) towards the target image. Masking this learned edit interpolation to only apply to one subject $S_{j}$ (“car”, “bird”) post hoc results in edits primarily affecting that subject.
Refer to caption: /html/2403.17064/assets/img/edit_delta_subject_specificity/a_blue_car_parked_in_a_parking_garage/full/0_000.jpg

#### 3.1.1 Subject Specificity of Semantic Edit Deltas.

We now investigate the relationship between these learned tokenwise semantic edit deltas $\Delta\mathbf{e}$ and the initial prompt.
At a high (semantic) level, our training method initially yields an adaptation of the full prompt embedding that substantially closes the gap between the originally generated images and the target images.
We find (see [Fig. 2](#S3.F2)) that only applying the subject-specific token edit $\Delta\mathbf{e}_{[S_{j}]}$ of the learned edit delta $\Delta\mathbf{e}$ suffices to obtain a mostly disentangled edit of $S_{j}$. This partial edit is semantically close to the full edit, affecting the rest of the image only minimally. Notably, we mask out the deltas only after training.
This implies that our training process associates semantic information to the specific token corresponding to each subject $S_{j}$ and that these token-specific deltas $\Delta\mathbf{e}_{[S_{j}]}$ are directly usable for influencing the subject’s attribute expression $\mathrm{expr}_{S_{j}}(\mathcal{A})$ on a semantic level. This provides a simple interpretable way to localize modifications (condition ii).

#### 3.1.2 Transferability of Semantic Edit Deltas.

Finally, we investigate the transferability of the learned $\Delta\mathbf{e}_{[S_{j}]}$ to other prompts. Instead of applying the subject-specific $\Delta\mathbf{e}_{[S_{j}]}$ to $S_{j}$ in the same prompt, we transfer deltas learned on one text/image pair to a new prompt. First, we test transferring a delta $\Delta\mathbf{e}_{[S_{j}]}$ learned on one subject $S_{j}$ (such as “woman”) from one prompt to the same word in a new prompt, as shown in [Fig. 3](#S3.F3)a. This results in a change of attribute expression $\mathrm{expr}_{S_{j}^{\prime}}(\mathcal{A})$ on the new subject $S_{j}^{\prime}$ semantically similar to the change seen on the original prompt (see [Fig. 2](#S3.F2)). These deltas can also be transferred to other subjects $S_{k}$ of a similar category (such as from “woman” to “man”, see [Fig. 3](#S3.F3)b). There, they result in similar changes to attribute expression $\mathrm{expr}_{S_{k}}(\mathcal{A})$ again but keep the underlying subject change (such as the gender change in this example) from the base text token intact. This shows that our previously defined condition iii) is effectively fulfilled.

Figure: Figure 3: We observe that learned subject edit directions (such as for “woman” on the left, from [Fig. 2](#S3.F2)) learned on one image/text pair can be transferred to other subjects. These can be transferred to novel prompts (unmodified images are marked in orange) that mention the same subject (a) or even ones that mention other subjects (b). In both cases, they cause the attributes (lip thickness, eyebrows, make-up, hairdo) of that subject to change in a similar direction.
Refer to caption: /html/2403.17064/assets/img/edit_delta_subject_specificity/close_up_portrait_of_a_woman_with_a_bird_on_her_finger/1/full/0_000.jpg

### 3.2 Identifying Specific Attribute Deltas from Contrastive Prompts

[Sec. 3.1](#S3.SS1) showed that subject-specific attribute modifications can be achieved by modifying the token(s) of the noun corresponding to the subject without requiring additional words. This implies that the CLIP text encoder likely already performs aggregation of semantic attributes into the corresponding subject, which is corroborated by []. Otherwise, the diffusion model would likely not learn to interpret the tokenwise embeddings this way. This is also corroborated by Li *et al*. [], who find that concepts will be present in multiple places in the tokenwise embedding instead of being localized to just its specific tokens. Despite adding attributes typically affecting a multitude of token embeddings, our previous findings indicate that just modifying the token $\mathbf{e}_{[S_{j}]}$ already suffices to enable substantial semantic changes.

Motivated by this finding, we propose identifying semantic directions in the tokenwise embedding space that affect specific attributes $A_{i}$ from contrastive prompts similar to [] (such as “a young person” vs. “an old person” for the “age” attribute, both prompts using the same noun for $S_{j}$).
We first obtain the tokenwise CLIP embeddings $\mathcal{E}_{\mathrm{CLIP}}(\mathrm{prompt}_{A_{i},+})$ and $\mathcal{E}_{\mathrm{CLIP}}(\mathrm{prompt}_{A_{i},-})$ for the positive and negative prompts, respectively. Then, we compute the difference between the token embedding of the subject in both embeddings

$$ $\Delta\mathbf{e}_{A_{i}}=(\mathcal{E}_{\mathrm{CLIP}}(\mathrm{prompt}_{A_{i},+}))_{[S_{j}]}-(\mathcal{E}_{\mathrm{CLIP}}(\mathrm{prompt}_{A_{i},-}))_{[S_{j}]}.$ (2) $$

This directly yields a direction that corresponds to the target attribute $A_{i}$. To obtain more robust estimates of this direction, we average it over a multitude of contrastive prompt pairs that describe the same concept.

##### Sampling.

During sampling, we simply add these learned deltas $\Delta\mathbf{e}_{A_{i}}$ to $\mathbf{e}_{[S_{k}]}$ of the target subject(s) $S_{k}$ with the desired scale $\alpha_{i}$. We then pass this modified prompt embedding to the diffusion model to generate an image causing no additional computational cost over standard sampling.
This yields similar behavior to adding adjectives such as “old” or “young” to the subject in the prompt but additionally enables control on a smooth scale while retaining subject-specificity.

Two examples of these learned directions are shown in [Fig. 4](#S3.F4). They illustrate that this approach works to identify directions that affect attributes such as a vehicle’s price but also often exhibits unrelated correlations such as car orientation or bike size.
As the CLIP text encoder is causal, this approach is also limited to attributes that can be described as a prefix to the target subject noun.

Figure: Figure 4: Two examples showing variations along the “vehicle price” direction identified using our CLIP embedding difference method ([Sec. 3.2](#S3.SS2)). Unmodified images are marked in orange. These directions successfully capture the target attribute and allow for fine-grained modulation but also show unwanted side-effects such as flipping the car’s orientation, which [Sec. 3.3](#S3.SS3) addresses.
Refer to caption: /html/2403.17064/assets/img/naive_clip_vs_learned/a_photo_of_a_car_price_0_naive_clip.png

### 3.3 Learning Robust Fine-Grained Attribute Deltas

Inspired by our findings from [Sec. 3.1](#S3.SS1), we introduce a method for targeted, fine-grained, subject-specific control of attributes in T2I generation. We previously found that editing token embeddings of a specific subject $S_{j}$ directly modulates its semantic attribute expression $\mathrm{expr}_{S_{j}}(\mathcal{A})$ and that these edits can be transferable.
Thus, we propose to leverage this approach to introduce fine-grained control into T2I models without having to modify them, that is, learning edit directions $\Delta\mathbf{e}_{A_{i}}$ to the embedded prompt $\mathbf{e}$ that directly correspond to subject-specific fine-grained modulations of the expression $\mathrm{expr}_{S_{j}}(A_{i})$ of a specific attribute $A_{i}$.

In general, to limit the $\Delta\mathbf{e}_{A_{i}}$ to apply to a specific subject $S_{j}$, we only modify the part of $\mathbf{e}$ that corresponds to $S_{j}$:

$$ $\mathbf{e}^{\prime}(\mathbf{e},\alpha_{i}\Delta\mathbf{e}_{A_{i}})_{[S_{j}]}=\mathbf{e}_{[S_{j}]}+\alpha_{i}\Delta\mathbf{e}_{A_{i}}.$ (3) $$

This modified embedding is then passed to the diffusion model in place of $\mathbf{e}$.

Figure: Figure 5: The same two examples as in [Fig. 4](#S3.F4), but with our learned edit deltas instead of our CLIP embedding differences. Unmodified images are marked in orange. Spurious correlations are reduced substantially.
Refer to caption: /html/2403.17064/assets/img/naive_clip_vs_learned/a_photo_of_a_car_price_0_ours.png

#### 3.3.1 Training

During training, we utilize the diffusion model’s world knowledge, specifically about which changes in the generated images $\mathbf{x}_{0}$ correspond to modulations of specific high-level attributes $A_{i}$.
We continue using contrastive text prompts that describe the target attribute $A_{i}$ to elicit a fine-grained direction in the model’s noise prediction space that corresponds to that attribute. We then backpropagate this direction through the diffusion model, distilling it into our delta $\Delta\mathbf{e}_{A_{i}}$. Like in [], this approach learns robust deltas $\Delta\mathbf{e}_{A_{i}}$ from just a set of contrastive prompts and does not require training images.

For each optimization step, we start by generating a random image $\mathbf{x}_{0,a}$ from the target category using a random prompt (such as “a photo of a person”) with standard sampling settings. This image will serve as the anchor for the optimization process. Starting from a noised version of this image $\mathbf{x}_{t,a}$ at a random diffusion timestep $t$, we then generate three predictions with the diffusion model: one prediction $\hat{\mathbf{x}}_{0,a}$ with the original anchor prompt and two predictions $\hat{\mathbf{x}}_{0,+},\hat{\mathbf{x}}_{0,-}$ with modified prompts. For those, added adjectives either increase (such as “a photo of an old person”) or decrease (such as “a photo of a young person”) the expression of the target attribute $\mathrm{expr}(A_{i})$ (such as age).

As is well-known, classifier-free guidance [] can effectively merge multiple such predictions into a new one, allowing for continuous application of different predictions. Starting from $\hat{\mathbf{x}}_{0,a}$, we can increase or decrease $\mathrm{expr}(A_{i})$ with

$$ $\hat{\mathbf{x}}_{0,\mathrm{target}}(\alpha_{i})=\hat{\mathbf{x}}_{0,a}+\alpha_{i}\cdot\left(\hat{\mathbf{x}}_{0,+}-\hat{\mathbf{x}}_{0,-}\right)$ (4) $$

Here, the guidance scale $\alpha_{i}$ controls both the direction (sign) and magnitude (value) of the attribute expression change.
We can then use these targets to train our semantic edit delta $\Delta\mathbf{e}_{A_{i}}$. Randomly sampling $\alpha_{i}$ during training allows us to reliably continuously modulate $A_{i}$ later on. When minimizing the difference between the diffusion model’s prediction and the target prediction, we then also adjust $\Delta\mathbf{e}_{A_{i}}$ by $\alpha_{i}$, leading to our delta training loss formulation:

$$ $\mathcal{L}_{\mathrm{delta}}=\mathbb{E}_{\alpha_{i}}\!\left[w(t)\left\|\hat{\mathbf{x}}_{0,\mathrm{target}}(\alpha_{i})-\hat{\mathbf{x}}_{0}(\mathbf{x}_{t,a}|\mathbf{e}^{\prime}(\mathbf{e},\alpha_{i}\Delta\mathbf{e}_{A_{i}}),t)\right\|_{2}^{2}\right].$ (5) $$

With a set of suitable prompts to describe increasing and decreasing expressions of a target attribute $A_{i}$, this objective learns robust attribute deltas $\Delta\mathbf{e}_{A_{i}}$.

During sampling, we use the same methodology for applying them as in [Sec. 3.2](#S3.SS2).
Compared to the results obtained using the CLIP text embedding differences (see [Fig. 4](#S3.F4)), these results (see [Fig. 5](#S3.F5)) also successfully capture the target attribute but exhibit fewer artifacts, such as the car’s orientation flipping and its age changing (see [Sec. 4.1](#S4.SS1)).

### 3.4 Global Correlations

Figure: Figure 6: Expression of Global Correlations with Different Sampling Methods: When changing the age of the woman (age delta on “woman”) starting from the prompt “A photo of a beautiful woman sitting on a sofa in her flat”, the background can be expected to change with her age. Using different sampling methods (b and c) substantially reduces these correlations (see, *e.g*., the sofa & plants on the shelves). Unmodified images are marked in orange. Delta scales $\alpha_{i}$ are identical across rows.
Refer to caption: /html/2403.17064/assets/img/disentanglement/woman_age_disentanglement_1_normal.jpg

Generally, image generation models learn correlations between different parts of the image.
As we do not modify the diffusion model itself, directly applying our learned edit deltas $\Delta\mathbf{e}_{A_{i}}$ to a specific subject $S_{j}$ causes the desired change in attribute expression $\mathrm{expr}_{S_{j}}(\mathcal{A})$, but additionally captures the entanglement between this subject’s attributes and the attributes of the rest of the generated image based on the diffusion model’s world knowledge. This means that this attribute change does not only apply to the part of the image $\mathbf{x}_{0,[S_{j}]}$ corresponding to $S_{j}$ but also the rest of the image (see [Fig. 6](#S3.F6)a). This helps to enable the generation of plausible images, as the entanglement is based on real-world dependencies. Modeling these correlations, however, might not always be desirable.

If the modeling of dependencies between the subject and its surroundings is not desired, we can augment our sampling method (without having to re-learn $\Delta\mathbf{e}_{A_{i}}$) during inference time.
Following [, ], simply not applying the edit delta for the first few steps of the diffusion generation process already substantially reduces the expression of these correlations at no additional computational cost. This especially helps to retain the original global image structure, as it is determined in the first steps of the generation process []. [Fig. 6](#S3.F6)b shows an example of this.
Since our model only alters the text embedding, we can also directly pair it with Prompt-to-Prompt [] to further improve appearance and structure disentanglement regarding the edit at the cost of doubling the inference cost. [Fig. 6](#S3.F6)c shows the corresponding example.

## 4 Experiments

Figure: Figure 7: Continuous Attribute Modifications. Unmodified images are marked in orange. (a) Variety of Attributes: Our learned edit deltas can capture a wide range of attributes and allow fine-grained control over their expression. All samples are generated using a delta scale from -2 to 2 without applying either of our two disentanglement strategies. (b) Zero-Shot Transfer: Our deltas can be learned on one model (SDXL) and transferred to others (including non-diffusion models) without re-training. (c) Standard P2P [] does not allow for continuous control starting from the original image without adjectives, only from inserted adjectives (“young”, “old”).
Refer to caption: /html/2403.17064/assets/img/single_attribute_reel/age.jpg

We evaluate our proposed method in a variety of settings, primarily on Stable Diffusion XL [], a standard off-the-shelf large-scale T2I diffusion model. We sample with a standard classifier-free guidance [] scale of $7.5$.
To test our method, we train a large variety of edit deltas for various different attributes, primarily focused on humans but also including vehicles and furniture.
[Fig. 7](#S4.F7)a shows a subset of these edit deltas.
Additionally, [Fig. 7](#S4.F7)b shows edit deltas learned on SDXL applied to other (diffusion and non-diffusion) models that use one of the two CLIP text encoders in a zero-shot manner.
Quantitatively, we evaluate on human attributes. We sample $25$ images per target subject noun, resulting in $100$ images for each attribute delta at each scale. In addition to the experiments presented in this section, we also investigate transferring edit deltas learned on one set of nouns to novel nouns in [Sec. F.2](#S6.SS2).

### 4.1 Learned Edit Deltas vs. CLIP Embedding Differences

First, we compare the attribute edit directions based on simple CLIP embedding differences ([Sec. 3.2](#S3.SS2)) with the learning-based ones ([Sec. 3.3](#S3.SS3)). For this comparison, we limit ourselves to a subset of edit deltas that can easily be described by prefixes in front of the subject (“young person”, not “person wearing a colorful outfit”). Here, the causality of the CLIP text encoder inherently limits the method based on embedding differences. Using only the applicable subset avoids this drawback and ensures a fair comparison of the methods.
Qualitatively, the directions obtained by taking differences of the CLIP token embeddings exhibit more unwanted side effects. Comparing [Fig. 4](#S3.F4) and [Fig. 5](#S3.F5), for example, the car flips its orientation in both directions in the difference-based setup.
To quantitatively evaluate the occurrence of undesired side effects, we assess the attained expression $\mathrm{expr}(A_{i})$ of the target attribute $A_{i}$ against the change in the image $\mathbf{x}$.
Following prior art [, ], we evaluate attribute expression change $\Delta\mathrm{expr}(A_{i})$ using the change of CLIP similarity of the generated image $I$ to a target prompt. As our attribute edit deltas are bi-directional, we compare to the prompt $\mathrm{prompt}_{+}$ that describes the positive direction (such as “old person”) and the prompt $\mathrm{prompt}_{-}$ for the negative direction (“young person”).
We compute this bi-directional relative CLIP score as

Figure: Figure 8: Attribute expression change (horizontal axis) against overall change in the image (vertical axis) as similarity/difference between the modified image at some scale and the reference image at scale 0 0. We compare learned edit deltas ([Sec. 3.3](#S3.SS3)) with CLIP text difference deltas ([Sec. 3.2](#S3.SS2)) at scales from -5 to 5 and find that our learned deltas allow the same attribute expression control at lower image deviation.
Refer to caption: /html/2403.17064/assets/x1.png

$$ $\mathrm{CLIP}_{Bi}(I)=\mathrm{cos}(\mathrm{CLIP}_{I}(I),\mathrm{CLIP}_{T}(\mathrm{prompt}_{+}))-\mathrm{cos}(\mathrm{CLIP}_{I}(I),\mathrm{CLIP}_{T}(\mathrm{prompt}_{-})),$ (6) $$

We then measure relative change to a reference image $I_{ref}$ without delta applied

$$ $\Delta\mathrm{CLIP}_{Bi}(I,I_{ref})=\mathrm{CLIP}_{Bi}(I)-\mathrm{CLIP}_{Bi}(I_{ref}).$ (7) $$

Comparing this score against the overall change in the image yields similar performance of learned and difference-based edit deltas in positive attribute expression directions ([Fig. 8](#S4.F8)). In the negative direction, however, the difference-based method struggles to successfully capture attribute expression without substantially altering the overall appearance of the image.

### 4.2 Global Correlations

To quantitatively evaluate the capturing of global correlations (see [Sec. 3.4](#S3.SS4)) when using different sampling methods, we use the same evaluation methodology as in [Sec. 4.1](#S4.SS1). Additionally, we use ArcFace-based [] person re-identification for edit deltas on human attributes to measure how much the person’s identity is affected by the attribute change. [Fig. 9](#S4.F9) shows that, for the same level of attribute expression, delayed application of the learned delta helps to substantially reduce unrelated changes in the image (especially as measured by LPIPS) at no additional cost during sampling. Combining our method with Prompt-to-Prompt [] further reduces these unrelated changes, but at the cost of doubling the sampling time.

Figure: Figure 9: Attribute expression change (horizontal axis) and overall change in the image (vertical axis) for our learned edit deltas ([Sec. 3.3](#S3.SS3)) with different sampling methods. For edit deltas targeting people, we also report person identity similarity (ReID).
Refer to caption: /html/2403.17064/assets/x3.png

### 4.3 Compositional & Subject-Specific Attribute Editing

In the general case, we find that our learned edit deltas are directly composable. This means that multiple edit deltas can be applied to the same subject, where their effects stack yielding fine-grained control over multiple attributes simultaneously as shown in [Fig. 10](#S4.F10)a, with a large number of additional samples shown in appendix [Sec. F.4](#S6.SS4). As they are subject-specific, multiple learned edit deltas can also be applied to different subjects (even ones of the same subject category) by applying each edit delta to just one of the multiple subjects mentioned in a prompt with individual scales as shown in [Fig. 10](#S4.F10)b and [Fig. 1](#S0.F1)a. Numerous additional examples are shown in appendix [Sec. F.3](#S6.SS3).

Figure: Figure 10: (a) Multiple attribute edit deltas can be composed simply by adding them. (b) Attribute deltas can be applied to different subjects with different magnitudes. Unmodified images are marked in orange.
Refer to caption: /html/2403.17064/assets/img/beautiful_man_stack_delay_smile_age_elegant.jpg

## 5 Conclusion

This work uncovers the powerful capabilities of the tokenwise CLIP [] text embedding for exerting control over the image generation process in T2I diffusion models. Instead of just acting as a discrete space of embeddings of words, we find that diffusion models are capable of interpreting local deviations in the tokenwise CLIP text embedding space in semantically meaningful ways. We use this insight to augment the typically rather coarse prompt with fine-grained, continuous control over the attribute expression of specific subjects by identifying semantic directions that correspond to specific attributes. Since we only modify the tokenwise CLIP text embedding along pre-identified directions, we enable more fine-grained manipulation at no additional cost in the generation process.

#### 5.0.1 Limitations & Future Work

This work is a step towards revealing the hidden capabilities of the text embedding input to common large-scale diffusion models and making them usable in straightforward ways. While our approach works for different off-the-shelf models without modifying them, it is also inherently limited by their capabilities. Specifically, our method inherits the limitation that diffusion models sometimes mix up attributes between different subjects. Complementary methods [, ] reduce these problems substantially, and future work could investigate their combination with our method in depth.

#### 5.0.2 Impact Statement

This work aims to improve the capabilities of text-to-image (T2I) diffusion models by introducing an efficient, simple-to-use method of influencing the expression of attributes of specific subjects in the generated images in a fine-grained manner. While, in general, similar targeted control has been possible before by utilizing editing-based methods on the original generated images, where generated images are inverted and re-generated with changes localized with custom edit masks, this method’s simplicity and efficiency potentially enables such a level of control for a wider audience. This, like other works in the space of improving the control over image synthesis models, carries the risk of further enabling the generation of harmful or deceptive content.

## Acknowledgement

We thank Timy Phan for proofreading and feedback.
This project has been supported by the German Federal Ministry for Economic Affairs and Climate Action within the project “NXT GEN AI METHODS – Generative Methoden für Perzeption, Prädiktion und Planung”, the German Research Foundation (DFG) project 421703927, and the bidt project KLIMA-MEMES. The authors acknowledge the Gauss Center for Supercomputing for providing compute through the NIC on JUWELS at JSC.

## F Additional Results

### F.1 Fine-Grained Control

Compared to Concept Sliders [], which enables fine-grained control over attribute expression on a global level (*i.e*., shared over all instances), our method allows targeting the attribute expression control to specific subjects by selecting the target subject via the prompt.
Compared to Prompt-to-Prompt [] (P2P), which provides subject-specific control like our method in [Fig. 11](#S6.F11), we find that our method offers substantially more fine-grained control over attribute expression when starting from the image generated from a prompt such as “a photo of a person”. For P2P, we start from the prompt “a photo of a beautiful man”, insert an adjective that describes the target direction (*e.g*., “a photo of a beautiful old man”), and then re-weight the adjective to control its expression(^3^33This approach is directly modeled after the one used in the official implementation at [https://github.com/google/prompt-to-prompt/blob/main/prompt-to-prompt_stable.ipynb](https://github.com/google/prompt-to-prompt/blob/main/prompt-to-prompt_stable.ipynb) to modulate the inserted word “fried” in the fried potatoes example and uses the same hyperparameters.). Here, it is obvious that the initial, coarse changes (such as from a neutral age to “old”) work well, but fine-grained modulations do not in the general case. Specifically, these re-weightings allow slight modulations around “old”, but they do not allow fine-grained control starting from the original image. This makes sense intuitively, as we previously observed (see [Sec. 3.2](#S3.SS2)) that attributes are aggregated in the subject by the CLIP text encoder and that the diffusion model uses these aggregated attributes. This means that simply reducing the weight of the adjective can not suffice to enable smooth interpolation between the original attribute expression (as it was in the prompt that did not contain any adjectives relating to the target concept) and the changed attribute expression, at least for models that use text encoders that exhibit this aggregating behavior.

Figure: Figure 11: Comparison of Capabilities for Continuous Attribute Modulation. We compare the capabilities of our learned deltas for continuous attribute modulation with that of Prompt-to-Prompt []. The unmodified image is marked in orange. Our samples are generated using attribute deltas being applied with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling). Those for P2P are generated starting from the same image by adding an adjective and then modulating its weight post hoc.
Refer to caption: /html/2403.17064/assets/img/appendix/p2p/img/p2p_age.jpg

### F.2 Subject Noun Transferability

We investigate how much the attribute deltas can generalize across different nouns that describe the same subject. We generally learn them on a set of different nouns that describe a subject of a specific category (*e.g*., deltas for people with the words “man”, “woman”, and “person”). However, these words typically do not cover the whole range of possible nouns that can be used to describe subjects of a general category. Ideally, one could learn one delta for one concept, such as age, on a small set of nouns and generalize across all nouns of a category or even to subjects of other categories.

First, we test the generalization of deltas learned for people on “man”, “woman”, and “person” and apply them to increasingly more specific nouns that describe people. Results are shown in [Figs. 12](#S6.F12) and [13](#S6.F13), and all prompts are “a photo of a beautiful <noun>”. As a baseline, we apply them to “child”, “mother”, and “father”, three words that are previously unseen but still describe very high-level sub-categories of people. We find that the learned deltas still work as expected. Similarly, for categories of jobs such as “doctor”, “barista”, or “firefighter”, which are substantially more specific and also substantially affect their clothing and the rest of the image, we find that they also work well. Finally, applying these learned deltas to very specific nouns such as the names “John” and “Jane” also works as expected. This demonstrates that these learned deltas can generalize well across a wide range of unseen nouns describing instances of a specific category, even if they were only learned on a small set of high-level, potential nouns.

Figure: Figure 12: Subject Noun Transferability. We stress-test applying deltas that have been learned only on the nouns “man“, “woman“, and “person“ to various other nouns that describe people. The unmodified image is marked in orange. All samples are generated using attribute deltas being applied with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/transferability/img/nouns_age.jpg

Figure: Figure 13: Subject Noun Transferability. We stress-test applying deltas that have been learned only on the nouns “man“, “woman“, and “person“ to various other nouns that describe people. The unmodified image is marked in orange. All samples are generated using attribute deltas being applied with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/transferability/img/nouns_width.jpg

### F.3 Multi-Subject Attribute Editing

[Figs. 14](#S6.F14), [16](#S6.F16), [17](#S6.F17) and [15](#S6.F15) show examples of modulating attributes in a subject-specific manner using our learned deltas. These show that various attributes can be applied to subjects individually, even if both subjects are of the same category (*e.g*., “people”). A slight correlation between, *e.g*., the age of the man and the age of the woman in [Fig. 14](#S6.F14) is visible and expected, as the diffusion model also models these dependencies between different subjects in the generated image. By applying both deltas with different strengths, the whole spectrum of combinations can be achieved.

Figure: Figure 14: Multi-Subject Attribute Modifications. The unmodified image is marked in orange. All samples are generated using one attribute delta each being applied to the two subjects mentioned in the prompt with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/multi_subject/img/person_age_person_age_2.jpg

Figure: Figure 15: Multi-Subject Attribute Modifications. The unmodified image is marked in orange. All samples are generated using one attribute delta each being applied to the two subjects mentioned in the prompt with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/multi_subject/img/woman_width_man_width.jpg

Figure: Figure 16: Multi-Subject Attribute Modifications. The unmodified image is marked in orange. All samples are generated using one attribute delta each being applied to the two subjects mentioned in the prompt with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/multi_subject/img/person_width_person_width_3.jpg

Figure: Figure 17: Multi-Subject Attribute Modifications. The unmodified image is marked in orange. All samples are generated using one attribute delta each being applied to the two subjects mentioned in the prompt with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/multi_subject/img/woman_age_man_age.jpg

### F.4 Compositional Attribute Editing

We show some 2d grids where two attributes are modulated for the same target subject in an additive manner in [Figs. 19](#S6.F19), [20](#S6.F20), [21](#S6.F21), [22](#S6.F22), [23](#S6.F23) and [18](#S6.F18). Both attribute deltas interact with each other according to the world knowledge of the diffusion model to produce a realistic image for every combination. This can especially be seen in [Fig. 20](#S6.F20), where the variant with increased age and makeup has substantially reduced wrinkles compared to the version with reduced makeup.

Figure: Figure 18: Compositional Attribute Modifications. The unmodified image is marked in orange. All samples are generated using two attribute deltas being applied additively with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/double_attribute/img/car_age_price_2.jpg

Figure: Figure 19: Compositional Attribute Modifications. The unmodified image is marked in orange. All samples are generated using two attribute deltas being applied additively with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/double_attribute/img/car_age_price.jpg

Figure: Figure 20: Compositional Attribute Modifications. The unmodified image is marked in orange. All samples are generated using two attribute deltas being applied additively with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/double_attribute/img/person_age_makeup.jpg

Figure: Figure 21: Compositional Attribute Modifications. The unmodified image is marked in orange. All samples are generated using two attribute deltas being applied additively with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/double_attribute/img/person_age_makeup_2.jpg

Figure: Figure 22: Compositional Attribute Modifications. The unmodified image is marked in orange. All samples are generated using two attribute deltas being applied additively with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/double_attribute/img/person_width_smile.jpg

Figure: Figure 23: Compositional Attribute Modifications. The unmodified image is marked in orange. All samples are generated using two attribute deltas being applied additively with a linear delta scale from -2 to 2 across each, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/double_attribute/img/person_width_smile_2.jpg

### F.5 Continuous Attribute Modification

To illustrate the breadth of attributes that can be modulated and how continuous the attribute changes are, we show a range of our learned delta attributes being continuously modulated. [Figs. 24](#S6.F24), [25](#S6.F25), [26](#S6.F26) and [27](#S6.F27) show examples where attribute deltas are applied with our delayed sampling, [Fig. 28](#S6.F28) shows attribute deltas applied for the full sampling time. For every category, we re-use the same sample instances as a starting point.

Figure: Figure 24: Continuous Attribute Modifications. Unmodified images are marked in orange. All samples are generated using a linear delta scale from -2 to 2, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/single_attribute/img/vehicle_age.jpg

Figure: Figure 25: Continuous Attribute Modifications. Unmodified images are marked in orange. All samples are generated using a linear delta scale from -2 to 2, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/single_attribute/img/person_age.jpg

Figure: Figure 26: Continuous Attribute Modifications. Unmodified images are marked in orange. All samples are generated using a linear delta scale from -2 to 2, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/single_attribute/img/person_elegant.jpg

Figure: Figure 27: Continuous Attribute Modifications. Unmodified images are marked in orange. All samples are generated using a linear delta scale from -2 to 2, with the deltas being applied after 10/50 steps (Delayed Sampling).
Refer to caption: /html/2403.17064/assets/img/appendix/single_attribute/img/person_makeup.jpg

Figure: Figure 28: Continuous Attribute Modifications. Unmodified images are marked in orange. All samples are generated using a linear delta scale from -2 to 2, with the deltas being applied for all steps.
Refer to caption: /html/2403.17064/assets/img/appendix/single_attribute_no_delay/img/person_long_hair.jpg

## G Implementation Details

This section gives details about the implementation of our method.
We generally use the default settings as set in diffusers(^4^44[https://github.com/huggingface/diffusers](https://github.com/huggingface/diffusers))-v0.25.0 with a classifier-free guidance [] scale of 7.5 and 50-step DDIM [] sampling unless specified otherwise.

### G.1 Main Method

#### G.1.1 Data: Contrastive Prompts

As the basis from which our edit deltas (both the learned ones in [Sec. 3.3](#S3.SS3) and the difference-based ones in [Sec. 3.2](#S3.SS2)), we use sets of contrastive prompts and prefixes (the same general setup as in [, ]). These are composed of multiple tuples of 3 prompts with a negative, neutral, and positive prompt each (*e.g*., (“young person”, “person”, “old person”) or (“weak woman”, “woman”, “strong woman”). These tuples generally include both multiple ways of expressing the attribute (*e.g*., “muscular”, “strong”, “body builder” for the “muscularity” attribute) and, if applicable, multiple ways of naming the target subject (*e.g*., “person”, “woman”, “man” for deltas targeting people). Additionally, we keep a list of prompt prefixes to achieve more variation. These span some aspects orthogonal to the target attribute (*e.g*., “a photo of a {$\emptyset$, mountain, road, BMX, folding, big, small, green, red, blue}” for the “age” attribute for “bike”). These variations of expressing the attribute and prefixes result in a large number of combinations for contrastive prompts, which, in turn, enables a more robust estimation of the underlying direction that modulates this target attribute in the prompt embedding.

#### G.1.2 Training ( Sec. 3.3 )

The tokenwise edit deltas $\Delta\mathbf{e}_{A_{i}}$ are implemented as learnable parameters of shape $1\times d_{\mathrm{CLIP}}$, with $d_{\mathrm{CLIP}}$ being the embedding dimension of the CLIP text encoder. For SDXL [], this is 2048. This delta is applied additively with scaling according to $\alpha_{i}$ to the target subject tokens (*e.g*., “person” in the case of “a photo of a strong person”) in the original text embedding $\mathbf{e}$. If the target subject consists of multiple tokens, we broadcast $\Delta\mathbf{e}_{A_{i}}$ across those tokens, although this is only rarely the case in practice.

We train our learned edit deltas $\Delta\mathbf{e}_{A_{i}}$ for 1000 steps at a batch size of 10. We use AdamW [] with a learning rate of 0.1, $(\beta_{1},\beta_{2})=(0.5,0.8)$, and weight decay of 0.333. All learned deltas are trained on a single A100 with 40GB of VRAM using a bfloat16 version of SDXL [].

For every entry in the batch, we use a random combination of prefix prompt and prompt tuple and sample an image with the neutral prompt and a random seed, stopping at a random timestep. We then compute the “vanilla” prediction starting from that step for all three prompts, resulting in $\hat{\mathbf{x}}_{0,a},\hat{\mathbf{x}}_{0,+},\hat{\mathbf{x}}_{0,-}$. In contrast to Gandikota *et al*. [], who use a similar approach, we sample our starting samples using standard sampling instead of a modified generation process.

We then sample four values for $\alpha_{i}\sim\mathcal{U}([-5,5]\setminus(-0.1,0.1))$ and compute $\mathcal{L}_{\mathrm{delta}}$ ([Eq. 5](#S3.E5)) using them. We found that sampling multiple values for $\alpha_{i}$ here boosts the performance of our learned deltas at little overhead cost (as the online sampling of the original images is the most costly part) and that values for $\alpha_{i}$ very close to zero were not particularly useful for the training process. Empirically, we find that most of our learned edit deltas are already close to convergence after 5 optimization steps, but we keep training for the full time for simplicity.

#### G.1.3 CLIP Embedding Differences ( Sec. 3.2 )

For the edit deltas determined via CLIP embedding differences, we apply the same idea as for the trained version ([Sec. G.1.2](#S7.SS1.SSS2)), but simply compute the difference between the subject token embeddings as in [Eq. 2](#S3.E2) and average them over the set of prompts.

#### G.1.4 Inference

During inference, we add the learned deltas $\Delta\mathbf{e}_{A_{i}}$ to the subject target tokens. The methodology here is the same as described in [Sec. G.1.2](#S7.SS1.SSS2).

##### Composition

When combining multiple deltas $\Delta\mathbf{e}_{A_{i}}$ during a single generation, we add each delta with the chosen scale to the chosen subject. Multiple deltas that are to be applied to the same subject are simply summed, making the results invariant to the order in which they are applied.

##### Sampling

We use three sampling variations in [Sec. 3.4](#S3.SS4). For the “Normal Sampling” version, we simply apply $\Delta\mathbf{e}_{A_{i}}$ during the whole inference process. For “Delayed Sampling”, we do not apply it for the first few steps (*e.g*., 10 for a 50-step sampling process) and then apply it, as done in []. For our combination with Prompt-to-Prompt [], we use a public reference implementation(^5^55[https://github.com/RoyiRa/prompt-to-prompt-with-sdxl](https://github.com/RoyiRa/prompt-to-prompt-with-sdxl)) and use the word replacement methodology to replace the original subject prompt embedding with our modified subject prompt embedding (original plus $\alpha_{i}\Delta\mathbf{e}_{A_{i}}$).

### G.2 Text/Image Pair Deltas

For learning full image deltas on a text/image pair, we use the same optimizer setup as in [Sec. G.1.2](#S7.SS1.SSS2) with AdamW [] with a learning rate of 0.1, $(\beta_{1},\beta_{2})=(0.5,0.8)$, and weight decay of 0.333. We learn deltas of shape $N\times d_{\mathrm{CLIP}}$, with $d_{\mathrm{CLIP}}$ being the embedding dimension of the CLIP text encoder and $N$ being the number of tokens in the tokenwise prompt embedding. We do not optimize the start-of-sequence and end-of-sequence tokens or the pooled embeddings in the case of SDXL. We train for 75 steps at a batch size of 1 and randomly select the noise level at each step.

### G.3 Evaluation

To compute perceptual image differences, we use LPIPS [] as implemented in the lpips(^6^66[https://github.com/richzhang/PerceptualSimilarity](https://github.com/richzhang/PerceptualSimilarity)) package with default settings at a resolution of $256^{2}$ (interpolated bi-linearly). For CLIP scores, we use the standard implementation in torchmetrics(^7^77[https://github.com/Lightning-AI/torchmetrics](https://github.com/Lightning-AI/torchmetrics)) (which outputs cosine similarities scaled to $[0,100]$) with default settings, including the default CLIP choice of the CLIP-ViT-L/14 trained by OpenAI []. For image-image similarity evaluations with DINOv2 [], we use the ViT-L/14 variant with registers [] and bi-linearly resize to $224^{2}$ before passing them to the model and comparing the cosine similarity of the CLS token outputs. Finally, for ReID evaluations, we use the ArcFace [] implementation provided by the insightface(^8^88[https://github.com/deepinsight/insightface](https://github.com/deepinsight/insightface)) python package with the default buffalo_l model, where we return the cosine similarity of the embeddings of the detected faces.

## H Image Copyright

[Fig. 2](#S3.F2) uses two photos obtained from Unsplash, one of a blue car in the garage(^9^99[https://unsplash.com/photos/a-blue-car-parked-in-a-parking-garage-Roq9jZmPetA](https://unsplash.com/photos/a-blue-car-parked-in-a-parking-garage-Roq9jZmPetA)) by Martin Katler and one of a woman with a bird on her hand(^10^1010[https://unsplash.com/photos/a-woman-with-a-bird-on-her-shoulder-vpIvmZBurgQ](https://unsplash.com/photos/a-woman-with-a-bird-on-her-shoulder-vpIvmZBurgQ)) (which is also used in [Fig. 3](#S3.F3)) by Ali Esfehaniyan. Both are licensed under the Unsplash license(^11^1111[https://unsplash.com/license](https://unsplash.com/license)). All other images shown in the paper are generated using Stable Diffusion XL [] unless noted otherwise.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The contrastive learning w.r.t. a subject with a prepended adjective $P_+$ is essential to the method. Do you need to manually select the generated images? For instance, do you need to verify if the image generated for "an expensive car" truly appears expensive, or do you let Stable Diffusion determine it automatically? Manually verifying each generated image related to $P_+$ could be highly costly.

2. In Eq.(3), $\lambda$ controls the magnitude of the modulation. Is it possible for $\lambda$ to exceed 1, thereby extending beyond the concept defined by the positive prompt? What is the range of $\lambda$? Any breaking point if $\lambda$ is too large?

3. In Fig.6, what would the distributions look like for negative $\lambda$?

4. The paper suggests that attributes can be adjusted linearly, and that by learning only the positive prompt, the model can naturally generalize to the negative prompt. How can this be explained?

5. How does the model perform in handling multiple complex attributes simultaneously, particularly in scenarios where attribute overlap is subtle?

6. Are there specific attributes or subject types where this approach performs less effectively, and how might these limitations be addressed?
soundness: 3
strengths: 1. The concept of leveraging token-level CLIP embeddings for fine-grained, subject-specific control is novel and enhances the flexibility of T2I models in generating nuanced images.
2. The paper provides a comprehensive methodology, with two approaches for direction identification that cater to different application needs. The learning-based approach, particularly, shows robustness and reduces unintended changes.
3. This approach complements current models without the need for model modifications, making it accessible and practical for integration.
summary: The paper presents an approach to attribute control in text-to-image (T2I) models, addressing a gap in subject-specific and continuous attribute modulation. This work identifies token-level "semantic directions" in the CLIP embedding space, allowing continuous, subject-specific adjustments. The authors propose two techniques to discover these directions: an optimization-free approach using contrastive prompts and a learning-based method employing backpropagation for robust direction identification. The solution integrates seamlessly with existing diffusion models, allowing intuitive and precise control over multiple attributes without modifying the T2I models.
weaknesses: 1. This approach is mostly heuristic, and the underlying reasons behind its effectiveness remain unclear.
2. The reliance on linear modulations for semantic directions may limit the flexibility of control in some cases.
3. The robust learning-based method requires backpropagation and multiple training instances, which may be computationally intensive, especially for real-time applications.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the weakness part.
soundness: 2
strengths: 1. The author identified a token-level attribute vector using a simple yet effective method, which allows control over the representation of specific attributes in the target generated by a diffusion model. 
2. Unlike other methods, this approach achieves fine-grained control over specific attributes of the target in the generated images by adding the vector to a specific word in the prompt.
3. Experimental results demonstrate the effectiveness of the proposed method and illustrate its transferability.
summary: This work advances the understanding of token-level directions within the commonly used CLIP text embeddings, demonstrating their efficacy for fine-grained, subject-specific control in text-to-image (T2I) models. The authors observed that diffusion models provide a smooth interpretation of CLIP text embedding vectors, allowing for the control of target attributes in generated images by identifying token-level semantic vectors within the CLIP space. Two primary methods are proposed. First, an optimization-free approach is introduced that contrasts embeddings from prompts that describe specific attributes (e.g., "a man" vs. "an old man") to determine the direction vector, with more accurate vectors obtained by averaging the differences across multiple prompt pairs. Second, a learning-based method is employed to identify attribute directions in the noise prediction space of the diffusion model. In summary, this work elucidates the effective application of token-level directions within CLIP embeddings, thereby enabling precise control over multiple attributes of individual subjects in image generation.
weaknesses: 1. The author derives attribute vectors through contrasting descriptions (e.g., "a man" vs. "an old man"). A more intuitive method would involve directly adding the embedding of "old" to "man," yet the author does not compare this simpler approach. It is advisable for the authors to compare their proposed method with this more intuitive approach.
2. While the authors mentioned that direct subtraction of CLIP vectors is only applicable to prefix attributes, the paper does not provide a clear explanation of how the optimization-based method addresses this limitation. Furthermore, the experimental results never examine related content, such as suffix attributes.
3. Will directly adding a vector to the corresponding word completely avoid affecting other targets? During image generation in diffusion models, text tokens interact with pixels through cross-attention layers, and in this process, an attention map is generated, which can reflect the degree to which image pixels focus on a particular word. It is possible that a pixel might exhibit high attention to multiple tokens, which may lead to overlapping influences on the image regions associated with different words. The authors are encouraged to consider such scenario and discuss it.
4. The paper mentions that experiments were conducted using SDXL, which has two text encoders; however, the author does not clearly specify how the attribute vectors are incorporated. Please provide a specific description of how the attribute vectors is incorporated in SDXL's dual text encoders. Additionally, since optimization involved training with images generated by the diffusion model, the model's performance could significantly affect the results. It is recommended that the authors investigate the performance vary across different diffusion models.
5. Based on the results presented in Table 1, the proposed method does not demonstrate exceptional performance across various metrics. Moreover, integrating other methods even leads to a decrease in performance on the subject-specificity metric. The authors are encouraged to discuss the potential reasons for such decreases in performance.
6. The results presented in the paper are based on a few simple attributes and prompts. To enhance the persuasiveness of the method, more complex scenarios should be demonstrated. For example, attributes could involve verbs like " jump" or cases where the target includes multiple words, such as "a red sports car". Additionally, visual comparisons with the baselines should be provided to offer clearer insights into the method’s effectiveness relative to other approaches.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: * The paper is well-written.

* The proposed method is straightforward and intuitive.

* The paper provides a good summary of existing methods and clearly articulates the motivation behind their work.
summary: The authors proposed a method to achieve fine-grained control over the attributes of images generated by diffusion models. They suggested introducing an adjective into the prompt and then comparing the embedding distances to derive a vector that can scale the attributes related to that adjective. This vector is then added to the embedding vector of the target subject. The authors also used synonymous versions of the adjective to obtain a more robust vector or optimizing to derive this vector.
weaknesses: * The novelty of the method is limited, and it lacks technical depth. I have some questions regarding the proposed method. First, it seems that the method may not be effective for all prompts. Consider the prompt "two people sitting on the lawn." How should one edit the attributes of just one person? Second, the method heavily relies on the selection of appropriate attribute adjectives, such as "old" and "young" for age. I am curious whether there are some attributes that are difficult to articulate. Intuitively, there are certain characteristics in nature that are currently challenging to describe accurately using human language.

* The experimental results do not support the authors' claims. In Table 1, the method' performance on the subject-specificity metric is weaker than that of the state-of-the-art method, e.g., AdapEdit, and even less effective than simply adding an adjective. Furthermore, when the authors' method is combined with other methods, it appears to degrade their performance. Given these results, I find it difficult to be convinced that the authors' method achieves a competitive performance. Additionally, I would like to know how the reported times in Table 1 are evaluated — do they include the time for training $\Delta e_{A_i}$? Based on my experience, I suspect they do not, which makes the cost comparison unfair. There is also no information on how many instances the metrics are averaged over. By the way, there are too few visual comparisons of different methods, and the authors did not provide results for editing multiple attributes.