Precise Parameter Localization for Textual Generation in Diffusion Models
Authors: Łukasz Staniszewski
Bartosz Cywiński
Franziska Boenisch
Kamil Deja
Adam Dziedzic
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
text edition
LoRA
localization
SD-XL
SD3
DeepFloyd IF
Primary area: generative models
TL;DR: We introduce a novel method for editing text within images generated by diffusion models, which modifies very few parameters and leaves the other visual content intact.

Abstract: Novel diffusion models can synthesize photo-realistic images with integrated high-quality text. Surprisingly, we demonstrate through 
attention activation patching that only less than $1$\% of diffusion models' parameters, all contained in attention layers, influence the generation of textual content within the images.  Building on this observation, we improve textual generation efficiency and performance by targeting cross and joint attention layers of diffusion models.  We introduce several applications that benefit from localizing the layers responsible for textual content generation. We first show that a LoRA-based fine-tuning solely of the localized layers enhances, even more, the general text-generation capabilities of large diffusion models while preserving the quality and diversity of the diffusion models' generations. Then, we demonstrate how we can use the localized layers to edit textual content in generated images. Finally, we extend this idea to the practical use case of preventing the generation of toxic text in a cost-free manner. In contrast to prior work, our localization approach is broadly applicable across various diffusion model architectures, including U-Net (e.g., SDXL and DeepFloyd IF) and transformer-based (e.g., Stable Diffusion 3), utilizing diverse text encoders (e.g., from CLIP to the large language models like T5). Project page available at https://t2i-text-loc.github.io/.

Published as a conference paper at ICLR 2025
PRECISE PARAMETER LOCALIZATION FOR TEXTUAL GENERATION IN DIFFUSION MODELS
Lukasz Staniszewskii & Bartosz Cywiński"
Warsaw University of Technology
{luks. staniszewsk wwwpwnswwwii.................. wwwp://sswiiiiemm.m..oommmmmmm.e
Franziska Boenisch
Kamil Deja
CISPA Helmholtz Center for Information Security
Warsaw University of Technology
foenisch(en c . de
IDEAS NCBR
kam 1 wewe.ddejjeee ja@eawwww.... edu. .p1
Adam Dziedzic
CISPA Helmholtz Center for Information Security adam. dz www.ddiiidiiissp......c
ABSTRACT
Novel diffusion models can synthesize photo-realistic images with integrated high-quality text. Surprisingly, we demonstrate through attention activation patch- ing that only less than 1% of diffusion models' parameters, all contained in atten- tion layers, influence the generation of textual content within the images. Building on this observation, we improve textual generation efficiency and performance by targeting cross and joint attention layers of diffusion models. We introduce several applications that benefit from localizing the layers responsible for textual content generation. We first show that LoRA-based fine-tuning solely of the localized layers enhances, even more, the general text-generation capabilities of large dif- fusion models while preserving the quality and diversity of the diffusion models' generations. Then, we demonstrate how we can use the localized layers edit textual content in generated images. Finally, we extend this idea to the practical use case of preventing the generation of toxic text in a cost-free manner. In con- trast to prior work, our localization approach is broadly applicable across various diffusion model architectures, including U-Net (e.g., SDXL and DeepFloyd IF) and transformer-
(e.g., Stable Diffusion 3), utilizing diverse text encoders (e.g., from CLIP to the large language models like T5). Project page available at https://22-eexxtooo.iibbbbb
INTRODUCTION
Recent advancements in generative models for the vision domain have demonstrated remarkable efficacy in image synthesis tasks and significant improvements in the quality and diversity of the generated outputs (DDPM (Ho et al.) 2020). LDM (Rombach et al. 2022 The next generation of models, including DeepFloyd IF (StabilityA] 2023). Imagen (Saharia al. 2022), Stable Dif- fusion 3 (SD3) (Esser et al. 2024). and FLUX (Labs|2024), extend this progress to photo-realistic generations with high-quality visual text. While introducing impressive capabilities, such models usually operate as black-boxes with complex architectures entangling various skills.
In this work, we propose shed some light the inner workings of recent diffusion models and introduce the first method to localize parts of the model responsible for the generation of textual content, based on activation patching technique (Meng et al. 2022). We determine that only 0.61% of Stable Diffusion XL (Podell et al 2024), 0.21% of Deepfloyd IF (Stability 2023). and 0.23% of Stable Diffusion 3 (Esser al. 2024) parameters are responsible solely for this task. Our observa- tions hold across various DMs' architectures, both U-Net and Transformer based, for DMs utilizing diverse text encoders, such as CLIP RRafford et al., 2021) and T5 (Raffel et al. 2020 Roberts al.) 2022). Additionally, we present several applications that benefit from our localization method.
Equal Contribution. Work done while authors were CISPA Helmholtz Center for Information Security.

---

Published as a conference paper at ICLR 2025
We first show that by selectively fine-tuning only the identified subset of layers responsible for textual content, we can significantly enhance the model's performance in generating text within images without reducing the quality and diversity of generated samples. Then, we present that by selectively applying patching, we are able to substitute the generated text without affecting other visual attributes of image. Our method does not require any additional extra data (potentially with human annotations), DM training (Brooks et al. 2023), semantic maps which indicate which part of images should be preserved during the diffusion process (Andonian et al. 2021 Tuo al.) 2024), or optimization. Finally, we extend our edition technique to prevent the generation of toxic text, on the fly without imposing additional computational cost.
Our contributions can be summarized as follows:
1. We localize small subset of cross and joint attention layers in diffusion models that deter- mine text generated within images. Our observations are archilecture-agnosti..
2. We introduce new fine-tuning strategy that targets only the localized subset of layers responsible for textual content, improving text generation performance while maintaining the model's overall generation diversity and efficiency.
3. We incorporate our findings into the new image-to-imant method for the text edition within synthetic images, outperforming previous techniques on standard benchmarks for image text editing, achieving superior accuracy and visual consistency.
4. We show that our method can also be effectively used to prevent the generation of harmful or toxic text within images in one generation pass.
BACKGROUND AND RELATED WORK
Text-to-Image diffusion models. Diffusion models (Song & Ermon 2020 Ho al. 2020) ap-
proximate data distribution by training a noise estimator cø(a,, f, y) to reverse the diffusion pro- cess. The synthetic images are then generated by sampling initial Gaussian noise, denoted as IT ~ ~N, I), and progressively removing the predicted noise at each time t 1 up until obtaining clean data sample zg- The noise predictor cØ ł, y) is usually implemented as a U-Net (Ronneberger et al. (2115) or, recently. (as in SD3 (Esser et al. (024)) transformer- bassdd model (Vaswani 2017 Peebles & Xie 2023). In common text-to-image DMs (Ramesh et al. 2022 Rombach et al., 2022 Saharia et al.) 2022, StabilityA] 2023), the conditioning input y is a text
embedding derived from textual prompt p using pre-trained text encoders, such as the text encoder
from CLIP (Radford et al. 2021) or the large language models like T5 (Raffel et al. 2020 as used in DeepFloyd IF (StabilityAl] or SD3 (Esser et al. 2024)).
Cross and Joint Attention layers. The integration of text conditioning into the denoising pro- cess is achieved through cross-attention layers (Vaswani 2017). The most standard cross-attention (used, e.g., in Stable Diffusion or SDXL (Rombach et al. 2022g) operates by computing three com- ponents: the query Q = AW the key K = eW' and the value V = eWV, where h and e represent the hidden image and text representations, respectively, and W W and W-V are learn- able weight matrices. The attention probabilities are then calculated using the following equation:
Attention (Q, K, V)- = softmax ( (QK" V, where d is a scaling factor equal to the dimension of the queries and keys. More recent diffusion models extend this mechanism further. Specifically, the DeepFloyd IF (StabilityA] 2023) model implements cross-altention layers where the keys and val- ues formed by concatenating the projections of both h and [sser al. (2024) further advance this mechanism by introducing a so-called joint attention, where each attention component (Q, K, and V) is a concatenation of projections from both h and e. Crucially, in this setup, both image and text projections are propagated throughout the diffusion model, in contrast to standard cross- attention layers where each attention block received the same static text-encoder embedding e as input. In our work, we demonstrate that our patching technique is invariant to these implementation changes and can be applied effectively across all of them.
Interpretability of diffusion models. Recent works have explored the inner workings of diffusion
models by analyzing cross-attention layers (Tang et al., 2023 Hertz et al. (2023). On the other hand, Park et al. (2024) explains the predictions of diffusion models at each denoising step using saliency
2

---

Published as a conference paper at ICLR 2025
maps. Other research efforts have focused localizing where specific concepts stored within diffusion models. For instance. Hintersdorf et al. (2024) pinpoint the memorization of individual training data samples within DMs the neuron level in cross-attention layers, using the z-score. Basu et al. (2024b) develop a framework utilizing causal tracing (Pearl, (001) to identify where knowledge of various styles, objects, or facts is stored within the Stable Diffusion model (Rom-) bach et al. (022). In follow-up work. [assu et al. (2024a) extend this framework by introducing mechanistic approach knowledge localization across different texl-to-image DMs. Despite being effective across models with standard cross atteenttntn implementations, such as Stable Diffusion XL (SDXL) (Podell et al., 2024) and DeepFloyd IF AtabiilityI (2023), it lacks analysis on the most recent attention variants, such as joint atlention (Esser et al. 2024). In contrast, our approach lo- calizes small fractions of components responsible for generating textual content and is applicable across different cross-attention variants.
Text rendering in diffusion models. Recent diffusion models, such as Stable Diffusion (Rom- bach al.) 2022), generate high-quality images conditioned on text prompts but often struggle with rendering coherent visual text. To address this limitation, more advanced DM architectures (e.g.,
SDXL, Deep Floyd IF, SD3 (Esser al., 2024)) and FLUX (Labs 2024)) incorporate multiple text encoders, often based on models like CLIP (Radford et al., 2021) or large language models like
T5 (Raffel et al. 2020), to enhance the quality of generated text within images.
In parallel with the above efforts, several other approaches have emerged to improve the fidelity of generated text by adding components to the generation pipeline. For example, TextDiffuser (Chen) itt al. 2023) employs a two-stage process where a layout transformer (Gupta et al. 2 first iden- lifies text coordinates as segmentation masks, which are later used to fine-tune latent diffusion model to accurately inpaint or modify text based on prompts. Similarly, AnyTText (Tuo et al.) 2024) integrates an auxiliary latent module to process inputs like text glyphs or masked images and a text embedding module using OCR to blend stroke data with image caption embeddings. Additionally, other works incorporate extra conditioning during generation, such as Zhang et al. (2024b) with sketch images or Yang et al. (2024), which leverages glyph instructions.
Fine-tuning diffusion models with LoRA. Low-Rank Adaptation (LoRA) (Hu et al. 2022) is fine-tuning approach known for its capacity to deliver high-quality results with both spatial and temporal efficiency. LoRA achieves this by introducing external low-rank weight matrices, which optimized for the attention layers of the base model while keeping the pre-trained model weights unchanged. After the training process, these low-rank matrices define the adapted model, which can then be applied to the target task. Recently, (Frenkel al. (024) introduced B-LoRAs that leverage LoRA to explicitly disentangle the style and components of an image. In our work, we tune the localized layers using LoRA to further improve the generated text within images.
Controlling diffusion models with cross-attention. In Appendix A we further describe related work on text-to-inn to-image models fine-tuning and image editing by leveraging cross-attention layers and manipulating the denoising steps through keys and values.
3 EXPERIMENTAL SETUP
Benchmark. For evaluation, we adapt two benchmarks from Yang et al. (2024) for the text edit- ing. SimpleBench consists of 400 prompts following the template 'À sign that says "<keyword>" "., while CreativeBench includes 400 more complex prompts adapted from GlyphDraw Ma et al. (2033), such as 'Flowers in a beautiful garden with the word "<keyword>> written.". The keywords used the benchmarks are from pool of single-word candidates from Wikipedia and categorized into four buckets based on their frequency: Bucket Bucket 10k Bucket and Bucket took- Both benchmarks contain the same set of keywords, which serve as text that should be generated in the images. In this work, we use 100 prompts from each benchmark, with words from Bucket lA as a validation set, and the remaining 300 prompts as a test set. The prompts from these benchmarks serve as the source prompts Ps. To create the target prompt pr for each ps, we use the same prompt template as in Ps. but select the keyword from a different source prompt, ensuring that the corresponding ps and pr differ only in the key kerwords.
3

---

Published as a conference paper at ICLR 2025
Metrics. We measure two main aspects of the generations. As text alignment, refer to the correspondence to the keyword provided in the prompt. As image alignment, we calculate the quality of the image outside of the modified text (e.g. background). To measure the text aligmment, we use the OCR F1 Score, which is calculated as follows: Fl Score = 2xPrecisson Recall Precision Recall : where Precision measures the ratio of predicted characters in the keyword, and Recall measures the ratio of characters in the keyword that are covered by the prediction. Additionally, we compute the Levenshtein distance (LD) between the keyword and the text predicted by the OCR model and CLIP-T Score (Radford et al. 2021) measuring the similarity of the target text (contained in the target prompt pr) and the text in the edited image. To measure the alignment between original and edited images, we calculate Mean Squared Error (MSE), which is the average squared difference between the reference and generated images, indicating how close the generated image is to the reference; lower values indicate higher similarity. We also compute a Structural Similarity Index Measure (SSIM) (Wang et al., (004)) that evaluates the perceived quality of generated image by comparing its luminance, contrast, and structure to reference image, with higher values indicating greater similarity. Finally, we use the Peak Signal-to-Nois Ratio (PSNR), which measures the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation, where the signal, in case, is the reference image and the noise is the error introduced by editing the image; higher PSNR values indicated greater fidelity.
Models. We identify the layers responsible for text generation in the three recent DMs, namely SDXL (Podell et al. 2024). DeepFloyd IF (StabilityAl 2023), and SD3 (Esser et al.) 2024). that differ significantly in their architecture, especially in the text encoder parts and the implementations of attention layers. To detect text in generated images, we use the EasyOCR model. We choose a non-multi-modal method for this task to ensure that OCR- based metries are computed purely based on the text present in images. We observe that multi-modal OCR models tend to guess the text based on the visual context, even when not present in the image. As a text detection model, we use the DBNet (Liao et al. 2020).
4 LOCALIZATION OF ATTENTION LAYERS RESPONSIBLE FOR TEXTUAL CONTENT GENERATION
We begin by presenting details of our patching technique for cross and joint attention layers, which we employ to localize the components of diffusion models responsible for the content of the gen- erated text. We demonstrate that our method generalizes across diverse model architectures despite differences in the implementations of attention layers and with different configurations as well as types of text encoders.
4.1 PATCHING TECHNIQUE
Recent works (Basu et al. 2024a Orgad et al. 2023) demonstrate that altering the key and value ma- trices of cross-attention layers can effectively influence the concepts generated by diffusion models. Specifically, Basu et al. (2024a) show that only certain attention layers within DMs are responsible for generating specific visual concepts, such as objects or styles. This approach that call injection is effective in U-Ner-based DMs such as Stable Diffusion or DeepFloyd IF, as shown in Figure [ (B). These models implement cross-attention layers that directly input the prompt embedding e and multiply it by the key W and value W matrices. However, it is unsuitable for the most recent DMs that leverage the joint attention mechanism (Esser et al. 2024), such as SD3 and FLUX. In these models, the subsequent attention layers process and modify both image and conditioning text, allowing each following layer to receive text embeddings modified by its preceding layers.
In our work, we leverage the activation patching technique (Meng et al. 2022) to identify the cross and joint attention layers responsible for generating text content in images across different DM's architectures. We present the overview of the patching process in Figure (A). Suppose we want to edit the text in the image is generated from the source prompt ps to match the text in the target prompt pr "A sign that says "tr"?. To measure the impact of each individual cross- cross-attnntion layer on the content of text generated in the output image, we first generate an image iz from pr, caching the keys Kς w and values V/2 ey W. (A.D, where er denotes the textual input part to the cross-attention layer. Then, while generating is from ps. overwrite Kş with Kr and Vs with Vy (A.II). We then calculate image and text alignment
A sign that says "ts"?
4

---

Published as a conference paper at ICLR 2025
(A.I) Text Prompt Caching
Localizing Injection
Target Prompt Py
Q
Text Hello"
Source Prompt Ps
K
Road sign with text
Text
V
V
"Singaapore it
Encoder
SINGAPORE
Eacoder
CT
Turget Prompt Pr
Cache
Text "Localize"
(A.II) Activation Patching
Without Injection
Q
Q
K
FOOD
.V
LOCALIZE
Souree Pronpt Ps
Corgl dog with sign saylng "food"
Encoder
HELLO
With Injection
Wath Pusching
Figure I: Overview of the localization process. Our goal is to edit the image generated from the source prompt ps using the target prompt pr. To find which cross and joint attention layers should be modified, we pass the target prompt pr through the DM, caching the keys and values. Then, while generating the image from ps we substitute the keys and values with the cached ones. We select the layers which yield the highest image and text alignment. (A) Localizing by Patching is applied to SD3, and (B) Localizing by Injection is used for SDXL and DeepFloyd IF.
# localized layers
total # of cross- -attention layers
# localized fraction of model
Model
parameters
parameters [%]]
SDXL (Podell et al. 2024
3
70
15.7M
0.61%
DeepFloyd IF (StabilityAl 2023
1
22
8.9M
0.21%
SD3 [sser et al. 2024
1
24
4.7M
0.23%
Table 1: Less than 1% of DMs' parameters influence text generation within the images.
metrics for the generations produced by the diffusion model with modified attention activations. To ensure consistency in our method, we always cache and overwrite only the text keys and values, which result from multiplying the textual parts of the residual stream by the key and value matrices. allows us to apply our technique across different DM architectures despite their differences in attention implementations.
4.2 CROSS-ATTENTION LAYEER LOCALIZZTIIN
We localize the layers responsible for text generation in three DMs with different architectures and text encoders: SIXXL, DeepFloyd and SD3. To this end, we run our patching approach for each cross-attention layer in each model on our validation set. As presented in the overview of the re- sults Table and Figure 2 we are able to successfully identify rross-attention layers that, when patched, cause the DMs to produce the text that closely matches the text in the target prompt pr. In both DeepFloyd IF and SD3 models, there is only a single layer that strongly responds when patched
SDXL
DeepFloyd IF
SD3
0.25
0.6
0.6
0.20
0.4
0.4
55
69
17
21
10
23
Index cross attention layer
Index of cross attention lay
Index of joint attention layer
Figure Localized attention layers responsible for the content of the generated text. We selectively patch individual cross and joint attention layers with computations for the target prompt and measure the responses with OCR Fl Score. We identify three layers with the highest responses in SDXL (55, 56, and 57), layer in DeepFloyd IF (17), and one layer in SD3 (10).

---

Published as a conference paper at ICLR 2025
OCR 1
SSIM ?
0.8
0.6
0.2
No layers injected
#1 layer injected
#3 layers injected
layers injected
#70 layers injected
YEATRE
SPONOORS
SPONOORS
SPONDORS
Figure 3: The localized layers effectively balance the text alignment with the target prompt pr and the image alignment with the source prompt ps. For ease of exposition, we measure the text alignment with OCR FI and the image alignment with SSIM. We observe that injecting the target prompt Pr to too many layers decreases the image alignment and introduces undesirable artifacts, e.g., the Japanese text on the robot's chest in 2nd image from the right and the lack of fish in the 1st image from the right. Conversely, injecting Pr to too few layers does not edit the generated text. We present more details about the experiment Appendix
Target prompt
Model
Tamplate
CLIP-T
Templatey
Texts:
OCR FI
Texty
SUMMEY
spring
spring
g
0.727
0.456
0,354
0.306
Template g lexty
lemplatey- (lostr
SDXL
0.724
0.430
0,203
0.331
Template Jexty
Doyd DF
6.71
0.453
0,554
0244
Template Textr
DeepRoyd IF
Template Texty
Desplloyd UF
0.721
0.465
0,275
0.452
Template g Testy
S03
0.675
0.443
0.544
0.231
BOYS
Template
0.599
0.443
Tomplate TT
Figure 4: Patching preserves visual components from the source prompt, taking only the tex- tual information from the injected target prompt. In all the combinations of templates and texts that we inject to localized layers of diffusion models (with other layers receiving both source tem- plate and source text), the final visual components of the image are always closer to the original template, while the textual content is always aligned with the one from an injected prompt. The source prompt is always defined ps==eemlatee::Texx T Templates: Texty, and Templatey: Text, (from left to right for the images).
while change the target prompts to
with the other prompt. On the other hand, in the SDXL model, we identify three such layers. The fact that in SDXL, the responses measured in the Fl Score are much more distributed than other analyzed models may be attributed to the fact that SDXL has significantly more cross-attention lay- ers than the other models and exhibits the lowest text generation capabilities. Overall, our findings suggest that a very small fraction of the DM's parameters is primarily responsible for the text content the generated images. Additionally, the successful localization of DM components across models demonstrates the applicability of our localization method across different DM architectures. In Fig- ure 3 we additionally visualize how patching a different number of layers affect the final generation in Stable Diffusion XL.
4.3 SPECIALIZATION OF THE LOCALIZED LAYERS
In the previous section, we localized layers that are responsible for the generation of the textual content. Here, we delve deeper into this analysis and evaluate their specialization. particular, we study what is the information extracted from the prompt by the selected layers and how it affects the generation. To measure this effect, we conduct a series of experiments with artificial prompts created as a combination of template that describes the background of the image and text, usually the form of a simple word. We present examples such prompts in Table

---

Published as a conference paper at ICLR 2025
We show that selected layers are only affected by the part of the target prompt that mentions the textual content. To that end, sample images with prompt ps Template : Texts used as conditioning for almost all the layers while patching the localized layers with of three target prompt options: (1) the same prompt (pT = Ps), (2) prompt that shares the same template but different text (pT Templates Textr) or (3) a prompt with different template and text (pr Template Texty). We present the result of this experiment in Figure 4 We observe that the final generation follows the text provided by the prompt pr used for patching. However, at the same time, changing the template in the target prompt does not affect the final generation, as the background image always significantly more aligned with the template from the source prompt. This observation means that the layers localized by our method are not only used for generating the textual content in the final sample but are also highly specialized, focusing solely on the textual content of the input prompt.
Table 2: Examples of prompts.
Template
Text
A book cover with text A sign that says paper letter with note
'Love'
'STOP"
'Lies'
5 APPLICATIONS OF OUR METHOD
Focusing on the localization of cross and joint attention layers for text generation offers several key advantages. In this section we highlight specific use cases where plays an instrumental role. We first show that we can precisely fine-tune selected layers to improve the quality of the generated text of a base model without affecting its remaining generative capabilities. Then, we present that with patching technique,
efficiently edit text from the model generations. We then extend the latter application to the cost-free technique for mitigating harmful or inappropriate text generation.
5.1 IMPROVING TEXT GENERATION THROUGH FINE-TUNING
We leverage our localization insights fine-tune pre-trained DMs on the task of visual text gener- ation. In particular, we show that by applying Low-Rank Adaptation (LoRA) only to the localized text-specific layers, we can significantly improve the quality of the generated text without affecting the model's performance on other tasks.
5.1.1 TRAINING SETUP
For training, we utilize a randomly chosen subset of 74,285 images from the MARIO-LAION 10M dataset (Chen et al. 2023). In order for the training text captions contain text that is directly presented on the corresponding training image, we construct them according to the template 'An image with text saying "<lext>" "' where "<lext>" constitutes of OCR labels corresponding the image. We compare the performance of applying LoRA to the localized layers with the baseline adaptation approach, for which we directly follow Hu et al.] (2022) and apply LoRA to all cross- attention layers. We optimize both models until convergence and evaluate the quality of model generations after the next epochs on our test set introduced in Section 3
To assess the quality of the generated text, we report OCR F1-Score and CLIP-T. Additionally, to quantify the effect of fine-tuning on the general generative capabilities of the model, we use the distribution precision and recall metrics (Kynkäänniemi et al., 2019) that measure the quality of individual samples (precision) and their diversity (recall) against the generations before fine- tuning. We adapt the original method to high-resolution generations from large diffusion models by substituting the original inception embeddings with the CLIP ones.
5.1.2 FINE-TUNING RESULTS
Our results demonstrate that by fine-tuning only the three cross-attention layers, identified as instru- mental for the generation of textual content, one can obtain model yielding higher-quality visual text compared to the model with all of the cross-attention layers fine-tuned while preserving the models' generation capabilities. As presented in Figure 5 (top left), even though fine-tuning of the whole model initially converges faster towards the higher performance, after 20 epochs of training, the model starts to overfit, what can be observed plcs presented in Figure 5 (bottom left). On the other hand, when fine-tuning selected layers, we can observe steady improvement in the quality of the generated text, with little effect on the model's
significant drop in the recall of generated sam-
7

---

Published as a conference paper at ICLR 2025
DC31 173
CLIPTT
Ours localized C-A layers
All C-A layers
Epoch
0:4
3303
Egoch 10
Reaal Dans 2ndosla CAless
loaal A.c-ll
Recall Precioon CLP scbeolings
1.50
1.5
Ejoch 20
Epoch 30
1per
Figure 5: Fine-tuning LoRA on localized layers improves text generation quality without com- promising overall generation capabilities. We apply LoRA fine-tuning to the SDXL model to enhance its text generation capabilities. (top left) The LoRA fine-tuning on the localized layers converges higher quality of the generated text (as measured by OCR F1 and CLIP-T metries). (bottom left) When fine-tuning LoRA on all cross-attention layers (denoted as C-A), the model quickly collapses, losing its ability generate examples that match the prompt. The diversity is significantly reduced, as indicated by recall. In contrast, fine-tuning LoRA only on our localized cross-attention layers prevents model overfitting while improving text generation quality. It pre- serves diversity while achieving higher fidelity measured by precision. (right) We also present this effect on sample generations. Longer LoRA fine-tuning (measured epochs) on localized layers improves text quality while preserving visual content, however, applying L to all layers results in significant degradation of the image quality and diversity.
generation performance and no visible mode collapse. Additionally, Figure (right) presents sam- ple generations from different training epochs, illustrating the changes to the base model induced by fine-tuning. We focus on LoRA for SDXL since this model has a significantly lower text generation quality than other studied DMs. We also present comparison between LoRA, the basic version of our method, and another editing method in Table The results indicate that our LoRA approach outperforms the other methods on all but two metrics. Overall, it achieves superior image and text alignment while preserving the fast execution time (from the basic version of our method).
5.2 EDITION OF GENERATED TEXT IN IMAGES
In this section, evaluate patching method leveraging the localized cross-attention layers in the task of text edition on images, where the goal is to preserve most of the source prompt-driven output while selectively modifying only the regions of the image where the source and target prompts editing framework (Hertz)
disagree. Our work can be directly compared to the prompt-to-prome let al. 2023) (denoted as P2P), where the image edition is controlled only by the text provided by the user. P2P also utilizes cross-attention layers in its design to modify visual concepts and defines SDXL,
target prompt, which derived from the source prompt. We evaluate both methods DeepFloyd IF. and SD3 models and present the results in Table 3 on our test set. Our standard patching method (denoted as "Ours") consistently outperforms P2P terms of image alignment to the source and text alignment to the target prompt. Additionally, our approach is significantly faster editing a single image, as reflected in the Execution Time column of the table.
While P2P is effective for DMs where the cross-attention layers' keys and values consist solely of text representations from the text encoder (such as SDXL), it struggles with models like DeepFloyd IF and SD3, where both text and image representations contribute to the keys and values. To address this, we introduce a modified version of P2P, denoted as P2P* for these models. Instead of overwrit- ing the entire keys and values during image generation, in the standard approach, P2P* overwrites only the textual components of the keys, allowing image elements to change. This modification en- ables effective text editing according to the target prompt, albeit with noticeable alterations
8

---

Published as a conference paper at ICLR 2025
Table 3: Our method outperforms P2P in text editing by generating higher-quality text while preserving the other visual components. We bold the best result for given DM in each metric.
Setup
Diftasion Model
Simpleleench
CreativeBench
Execution Time [s] 1
Image alignment
Text alignment
Image alignment
Text aligmment
MSE.
SSIM 1 PSNR
OCR FI
CLIP-T 1 LD4
MSE. 4. SSIM f PSNR
OCR
CLIP-T 1
Ours (t,, 50)
SDXL
44.78
0.80
32,09
0.34
0.78
75.95
25.34
0.89
35.06
0.32
0.52
102.85
10.37 4.25
Ours (f, 46)
SDXI.
0.81
32.25
0.34
0.77
75.45
23.49
0.90
0.32
0.82
102.79
10.37 1.
Oas LoRA
17.63
0.90
36.38
0.43
0.77
26.24
22.83
37.47
0.33
0.77
38.31
10.37 4.25
SDNL
30.77
0.29
0.69
75.72
57.26
30.93
0.26
0.78
99.50
31.17 4.
Oar 60)
DeepFloyd IF
73.15
29.70
0.70
0.800
10.65
57.92
0.71
$1.05
0.47
0.54
13.57 4.04
Ous
48)
Deepfloyd IF
0.70
10.85
53.50
0.54
13.57 1:04
DeepFloyd
105.60
0:41
0:27
10.23
0.61
28.04 1.78
P2P
Deephloyd
0.21
27,91
13.48
96.85
0.11
0.62
4.20
Oar
0.68
0.76
4.96
0.74
15.23
Our
29.54
0:53
0.70
5.79
30.61
0.41
0.75
15.23
SDS
0.82
28.65
0.57
82.53
0.82
29.13
0.29
0.71
00.55
118.30 15
P2P
SD3
98.22
0.58
28.24
0.90
0.88
2,06
85,77
0.64
28.90
0,66
0.90
62.59
118:30 55
the source image. Furthermore, in our visual text modification approach, the target prompt pr can differ from the source prompt ps in the prompt length and positions of tokens representing the text to change, as opposed to the P2P approach. In the Appendix K present example edition results for our localization-balt baasdd text edition method. In particular, we show that we can modify texts of varying lengths with our method.
5.3 PREVENTING GENERATION OF TOXIC TEXT
We observe that diffusion models, even the ones equipped with safeguards against generating NSFW (Not Safe For Work) content, tend to simply copy-paste the text from the prompt to the image. As a result, while the visual content may be safe thanks careful filtering of the fine-tuning dataset, the text in the generated images can still be harmful. We carry out experiments on known methods, such as Safe Diffusion (Schramowski et al., 2023) and Negative Prompts (Max Woolf, 2022), to evaluate their effectiveness in preventing the generation of toxic content and find out that those methods underperform. To address this issue, we propose new approach the application of our edition technique to prevent the generation of toxic text within images.
Our goal is to address scenarios where a model provider exposes a diffusion model for generating images from textual prompts. In this setting. a user may submit source prompt Ps containing toxic textual content intended to appear in the generated image. Detecting toxicity in the images is crucial for online platforms to enforce community guidelines and remove inappropriate material. With advancements in large language models, toxic text can be reliably identified (Zhang et al. 2024a) and rephrased to ensure that the generated image suppresses harmful content. To achieve this, the toxic portion of the source prompt is replaced with a non-harmful text or a placeholder sequence, such series of stars (*).
We harness our precise localization of the cross-attention layers responsible for generating textual content in images to prevent the model from outputting harmful text. In particular, the prompts identified as toxic are substituted with a non-harmful text on the fly using our patching technique. This allows us to remove the toxic content from the final generation without altering the remaining visual content. We achieve this result with a single pass through the diffusion denoising process without imposing any additional computational cost.
In Table we compare our method with three baseline techniques. First, we leverage negative prompting. It was suggested (Max Woolf (022) that the generative process can be more effectively guided by using negative text prompts that instruct diffusion model to exclude specific elements from its generated images. In that approach, we set the negative prompt to 'text "<word>> "' where <word>> is a harmful word from ps. We also run Safe Diffussion (Schramowski et al.] 2023) on safe prompts, which works by intervening directly in the latent space of diffusion models to remove and suppress inappropriate content during image generation. Additionally, introduce Safe Dif- fusion where we adapt the method (its safe prompts) to the task of toxic language removal. We present the details of adaptation in Appendix
9

---

Published as a conference paper at ICLR 2025
Table Our method can be used prevent the generation of toxic text in images. We bold the best result for given DM in each metric and the runner-up is underlined.
Method
Diffusion Model
MSE +
SSIM 1
PSNR 1
OCR F1 +
Toxicity score +
Ours
SDXL
48.20
0.79
31,68
0.20
0.003
Negative prompt
SDXL
77.95
0.71
31.76
0.23
0.052
Safe Diffusion
SDXL
49.46
31.33
0.34
0.222
Safe Diffusion*
SDXL
49.41
31.33
0.209
Prompt Swap
SDXL
79.41
31.65
0.000
Ours
DeepPloyd
74.96
0.61
29.60
0.32
0.018
Negative prompt
DeepPloyd
100.50
28.12
0.250
Safe Diffusion
DeepFloyd IF
64.30
30.19
0.79
Safe Diffusion*
DeepPloyd IF
63.65
0.74
30.25
0.79
0.540
Prompt Swap
DeepPloyd IF
100.99
0.35
28.10
0.30
0.015
Ours
SD3
72.61
0.70
29.72
0.018
Negative prompt
SD3
101.63
0.53
28.08
0.77
0.407
Safe Diffusion
SD3
34.99
0.86
34.25
0.73
0.571
Safe Diffusion
SD3
33.67
0.87
34.56
0.73
0.568
Prompt Swap
SD3
98.58
0.51
28.22
0.30
0.015
our approach, we replace the toxic word in the source prompt ps with a non-harmful suggestion and form the target prompt Pr. We also include potential method, that, similarly to us, is based on prompt edition, which we refer to as Prompt Swap. In this method, we apply the LLM-rreprased non-toxic prompt to the entire diffusion model instead of doing only for our localized layers.
In the experiments, each of the source prompts ps (we use 400 in total) contains a harmful word from LDNOOBW (2020). We obtain the edited generations from each approach, the OCR on the output images, and for the text returned from OCR, we calculate the toxicity score using the RoBERTa-based classifier (Liu al., 2022). We show that Negative Prompt and Safe Diffusion (in both versions) methods are incapable of removing toxic textual content from generated images. For Prompt Swap, we observe that this method marginally outperforms our approach in toxic text prevention. However, the introduced change in the modified prompt strongly impacts other visual aspects of image, which is not the case for our solution.
In the Appendix G we argue that preventing the change of visual attributes, even when the end user did see the original image, is important order to, i.e., preserve the emotions expressed in the original prompt to the model. We demonstrate that our approach successfully substitutes toxic text from the generated images without significantly altering the remaining part of the generation, mak- ing it the most reliable solution. We include example generations and detailed evaluation supporting this claim in Figure 12
CONCLUSIONS
This work identifies critical cross and joint attention layers diffusion models that directly influence text generation within images. Our proposed patching method adaptable various diffusion model architectures, regardless of the text encoder used. We demonstrate that in SDXL, only three layers (out of 70) impact text generation, while DeepFloyd IF and SD3, only a single layer is responsible for the generated text (out of 22 and 24, respectively). Fine-tuning these localized layers using LoRA significantly improves the quality of the generated text a base model without affecting its remaining generative capabilities. This selective targeting approach also increases the efficiency and precision of image-editing methods applied to text, reducing unintended modifications to non- textual visual elements. Additionally, our method can be leveraged create effective safeguard against the generation of harmful or toxic text in images, further highlighting its practical utility in safer and more efficient text-to-image generation workflows.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The project was funded by German Research Foundation (DFG) within the framework of the Weave Programme under the project titled "Protecting Creativity: On the Way to Safe Generative Models", funding number 545047250. The project was also supported by the National Science Centre, Poland, grants no: 2023/51/B/S66/000044 and 2022/5/BB/ST60281 7, PLGrid grant no. PLG/2024/017266, and by the Warsaw University of Technology within the (IDUB) programme.
REFERENCES
Alex Andonian, Sabrina Osmany, Audrey Cui, YeonHwan Park, Ali Jahanian, Antonio Torralba, and David Bau. Paint by word, 2021.
Samyadeep Basu, Keivan Rezaci, Priyatham Kattakinda, Vlad I Morariu, Nanxuan Zhao, Ryan A Rossi, Varun Manjunatha, and Soheil Feizi. On mechanistic knowledge localization in text-to- image generative models. In Forty-first International Conference on Machine Learning, 2024a.
Samyadeep Basu, Nanxuan Zhao, Vlad Morariu, Soheil Feizi, and Varun Manjunatha. Local- izing and editing knowledge in text-to-image generative models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- wwtp:oouuuuuueeeoem -d ne 6S0Q
view.net, 2024b. htttptt:p///opeeeeee
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402, June 2023.
Jingye Chen, Yupan Huang, Tengchao Lx, Lei Cui, Qifeng Chen, and Furu Wei. Textd- iffuser: Diffusion models as text painters. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 9353-9387. Curran Associates, Inc., 2023. URL https:
http::://proceednngs..e....
cc/ pwpprpappr...ee soarr_ffleee www.ss/ppppr//////////ee..// file/
paf
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content separation using b-lora, 2024.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://gpenrevier net ottp///oooummdooooopppp
Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layouttransformer Layout generation and completion with self-attention. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1004-1014, 2021.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Con- ference on Learning Representations, 2023. URL https:/ https:///penzeeiew wwwpreveeneeeeeeeeeeepeeee net forum?
id= CDixzkzeyb
Dominik Hintersdorf, Lukas Struppek, Kristian Kersting, Adam Dziedzic, and Franziska Boenisch. Finding nemo: Localizing neurons responsible for memorization in diffusion models. arXiv preprint arXiv:
2024.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. In Confer- on Neural Information Processing Systems (NeurIPS), pp. 6840-6851, 2020.
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representations, 2022. URL bectps://operre/e// nnt/
id=nZeVKeeFYf9
Timothy Jay and Kristin Janschewitz. The pragmatics of swearing. 2008.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. CVPR, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Black Forest Labs.
Flux.1, 2024. bl ack forest- forest-labs/
URL https://blcckforettlaas ai/
announci nn-bllg-b
LDNOOBW.
List of dirty,
obscene and otherwise bad htttps://github. com BtomLLDNNOBB/ wwweeeaaaand-heeewie-eeeeeeeeeee Words
naughty,
words:
otherwise-bad-woods,
2020.
URL
Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. Proceedings of the AAAI Conference Artificial Intelligence, 34(07):14744-14493 1481, Apr. 2020, doi: 10. 1119666/a...////// w34107.6111111 URL BRLLhttps://// /tctts::/ojs.... aaai. atg/index..php/Aaae
6812
Shu Liu, Kaiwen Li, and Zuhe Li. A robustly optimized bmre for aspect sentiment triplet extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 272-278, 2022.
Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, and Xiaodong Lin. Glyph- draw: Seamlessly rendering text with intricate spatial structures in text-to- text-to-image generation. arXiv preprint arXiv:2303. 17870, 2023.
Max Woolf. Negative Prompts in Stable Diffusion, 2022. URL https::/minimatt 2022/1//taaleeddlffissooooo www.ioonnnnntiiiiieeeeee.. prompt/ Accessed: November 26, 2022.
com/
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372 2022.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7053-7061, 2023.
Ji-Hoon Park, Yeong-Joon Ju, and Seong-Wie Whan Lee. Explaining generative diffusion models via visual analysis for interpretable decision-making process. Expert Systems with Applications, 248: 123231, 2024.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localiz- ing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 23051-23061, October 2023.
Judea Pearl. Direct and indirect effects. Proceedings of the Seventeenth Conference on Uncer- tainty in Artificial Intelligence, UAI'OL, pp. 411-420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Quynh Phung. Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. Computer Vision and Pattern Recognition, 2023. doi: 10.1109/CVPR52233.22224 00758.
12

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for reshuresolttion image synthesis. In The Twelfth International Conference Learning Representations, 2024. URL Forum?id=da
net 1 f
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Interna- tional Conference on Machine Learning (ICML), pp. 8748 8763, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 1-70 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIPP latents. arXiv preprint, arXiv:2204.06125, 2022.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bu- lian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:220..17189, 2022. URL lattps://ant //t::i//arvv.... org/ abs 2203. 17189
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- image segmentation. Medical Image Computing and Computer-Assien Assisted Intervention (MIC- CAI), pp. 234-241, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Den- ton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Sali- mans. Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorcalistic text-to- image diffusion models with deep language understanding. hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 36479-36494. Curran Associates, Inc., wtwppeeeiilleeeppee////// file #ile:c///SSeecddeo//2222223bbbb org.apaeeeeeteteaeeccccceoeeeeee Confereeccn
In S. Koyejo, S. Mo-
2022. URL https://pproceeddss...ee.. neurips cc wttp:/pppppppppeeeeeeee
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 22522-22531, 2023.
Sefik Ilkin Serengil and Alper Ozpinar. Hyperextended lightface: A facial attribute analysis frame- work. In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pp. 1-4. IEEE, 2021. doi: 10.1 1,.10999CCET7533422220010 96556697. URL htt.tht://eeeeeplllllle leee org/ http://dcmmnntttt55e0
Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models.
In Conference on Neural Information Processing Systems (NeurIPS), pp. 12438-12448, 2020.
StabilityAI. DeepFloyd IF: novel state-of-the-ane open-source tex--e----te----teeea- model with high de- gree of photorealism and language understanding. hitps: htttp:://gituu hwtphbt.ttuugg.g com/ cwm/deep floyd/ TF 2023. Retrieved 2023-04-17.
13

---

Published as a conference paper at ICLR 2025
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stene- torp, Jimmy Lin, and Ferhan Türe. What the daam: Interpreting stable diffusion using cross attention, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). pp. 1921-1930, June 2023.
Yuxiang Tuo, Wangmeng Xiang, Jun- Yan He, Yifeng Geng, and Xuansong Xie. Anytext: Multi- lingual visual text generation and editing. In The Twelfth International Conference on Learning Representations, 2024. URL https :/ htt///pprrevviee
net/ F
Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, and Kai Chen. Glyphcontrol: Glyph conditional control for visual text generation. Advances Neural Informa- tion Processing Systems, 36, 2024.
Jiang Zhang, Qiong Wu, Yiming Xu, Cheng Cao, Zheng Du, and Konstantinos Psounis. Efficient toxic content detection by bootstrapping and distilling large language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Fighth AAAI Conference Ar- tificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances Artificial Intelli- gence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 2024a. doi: (0.1609/AAAI V38119.3017..
AAAI Press, URL bttp:::/60..org/100..B8
v381 30178
Lingjun Zhang, Xinyuan Chen, Yaohui Wang, Yue Lu, and Yu Qiao. Brush your text: Synthesize any scene text on images via diffusion model. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educattinnll Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 7215-7223 AAAI Press, 2024b. doi: 10. AL.
Natarajan (eds.), Thirty-Eighth AAAI Conference
URL
sttps://dle org 0w000...11199/0 aaaii w18...
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK ON MANIPULATING DIFFUSION MODELS WITH GROSS-ATTENTE
Recent works introduced methods that leverage cross-attention layers for better control of text-to- image models. In Kumari et al. (2023), the authors present an efficient way of customization of text-to-image diffusion models by fine-tuning a subset of cross-attention layer parameters. While their approach demonstrates that targeting the key and value matrices in all the cross-attention layers sufficient to introduce new concepts, we reveal that tine-tuning those matrices in fewer than 5% of cross-attention layers (see Table I enables better quality of the generated text.
Geyer et al. (2024) presents a framework that enables video editing using text-to-image diffusion models. Specifically, the authors introduce method of editing the keyframes by extension of self- attention layers in which the keys from all timeframes are concatenated in order to encourage the frames to share global appearance. The presented solution offers effective approach to the semantic video edition.
Prompt-Mixing (Patashnik al. 2023) enables to explore different shapes of objects in image. In order for objects to stay in the same positions but change their appearance, the method operates in the inference time and, in different denoising timestep intervals, injects different prompts into the cross-attention layers. In our work, we use similar injection mechanism that we apply only the selected text-contoolling layers. We evaluate the effect of injection at different denoising steps in the Figure
Cross-Attention Refocusing (Phung et al., 2023) is a calibration technique enabling better attend- ing of tokens representing objects to image regions. By performing multiple intermediate latent optimizations by using CAR loss and Self-Attention Refocusing loss, authors achieve better con- trollability of the layout of generated objects. Similar to our work, CAR focuses on cross-attention maps but aims to strengthen attention to the correct token while reducing it elsewhere.
Plug-and-Pin (Tumanyan et al., 2023) is an effective image-to-image translation method. In this work, the authors show that in the denoising procedure, one can extract spatial features from the U- Net decoder's 's Residual Blocks and their following self-attention layers, obtaining encodings of the composition of the image. Next, by passing a different prompt during the denoising procedure for the same initial Gaussian noise, differing in image attributes specified in the condition. In this work, we show that by focusing on text-related features we can perform precise edition by targeting single attention layer.
inject previously extracted features and obtain generations,
B SELECTION OF DENOISING TIMESTEPS
To further refine the identification of text generation capabilities in DMs, investigate from which point in the diffusion denoising process the key and value matrices should be patched to achieve the highest performance in text editing. We present the results of this analysis in Figure 6 We observe that when starting the patching from the later timesteps we can observe better preservation in the visual attributes of modified image and improve the quality of the generated text, increasing its similarity to the text from the target prompt pB- This trend aligns with the work by Hertz et al. 2023), where authors show that only the overall structure of an image generated steps of the diffusion denoising process. Thus, in order to reduce the change in visual attributes, apply our patching method to localized attention layers starting from timesteps: t, 46 for SDXL, te 26 for SD3, and f, 48 for DeepFloyd IF. Attention activations from timestep T to t, remain unchanged while we patch all activations from timestep f, to
the initial
15

---

Published as a conference paper at ICLR 2025
HSE
5D3 start patching timestop p- Image aligumeen
SD3 start patching timostop text allmmment
Medte
02
(a) Image alignment vs Diffusion Patching Timestep SD3.
(b) Text alignment vs Diffusion Patching Timestep SD3.
352 vip
SoDE 7000 xto
DoopFlovd start patching timestep image alienment
DeepFlord patching timestep text alkemment
0040
45
Star
(c) Image alignment vs Difffusion Patching Timestep DeepFloyd IF.
(d) Text alignment vs Diffusion Patching Timestep DeepFloyd IF.
patching timestop image alignment
SDXL patching timestep algenment
Monte
S0.rh polchitg Canddr
Start patcinn
(e) Image alignment Diffusion Patching Timestep SDXL.
(I) Text alignment Diffusion Patching Timestep SDXL.
Figure 6: Starting the text edition from a later diffusion timestep improves both image and text alignment. We analyze the impact of the diffusion timestep from which we start the patching on the image and text alignment. We observe that find an optimum diffusion timestep that can simultancously improve image and text quality.
C LORA FINE-TUNING ACROSS DIFFERENT SETUPS
To further strengthen the evidence that have correctly identified the cross-attention layers sponsible for the content of the generated text, we conduct the LoRA fine-tuning process on other sets of three cross attention layers. These sets selected based on the OCR F1 Scores presented in Figure - specifically, we select three sets of adjacent layers with the highest and lowest sum of F1 scores, respectively. As shown in Figure 7 observe significant performance gap between the fine-tuned layers we localized and any other set of layers. Notably, some of the chosen layer sets even decrease performance compared to the base SDXL model.
Lacaland
Hipart3
OCR
CLIP-T
0.5
0.53
0.55
0.0
0.10
2016
8.0
044
301
100
13
Egooc
Figure 7: LoRA SDXL Fine-Tuning Across Different Setups. We fine-tune LoRA applied to the SDXL model to improve the text generation capabilities of the base model. When we fine-tune LoRA on all cross-attention layers, the model quickly collapses and loses its ability to generate ex- amples that match the prompt. In contrast, when fine-tune LoRA only our localized three cross-attention layers, we successfully prevent model overfitting while also improving text genera- tion quality. This trend is not observed when apply LoRA other sets of three layers.
16

---

Published as a conference paper at ICLR 2025
D LORA FINE-TUNING WITH DIFFERENT TRAINING SET SIZES
To evaluate how findings from Section 5.1 generalize to varying training sizes, fine- tune LoRA applied to the SDXL model on datasets ranging from 20k to 200k samples. To mitigate potential overfitting, especially in configurations where LoRA is applied to every cross-attention layer (Full model setup), we scale the training set size up to 200k samples. We train each setup for 12k steps with batch size equal 512 and learning rate of le-6.
In Figure [] we plot the recall and precision metrics across training steps. Notably, even with a substantially larger dataset in the Full model 200k configuration, the model exhibits similar collapse to what is observed when training on smaller subsets. Morcover, both recall and precision remain largely unchanged across different setups, demonstrating the robustness of our approach, which focuses on fine-tuning specific layers.
Additionally, in Figure g we plot the OCR F1 Score and CLIP-T metrics, highlighting that fine- tuning localized layers, even with as few as 20k samples, results in better performance than the Full model setup trained with 200k samples.
Ours 20%
Ours 50k
Ours 70k
Ours 150k
Ours 200%
Full model 74k
Full model 200k
Recall
Precision
1.0
1.00
0.9
0.95
0.8
0.90
0.7
0.85
0.6
3000
6000
9000
12000
3000
6000
9000
12000
Training Step
Training Step
Figure 8: Scaling up training size when fine-tuning all cross-attention layers does not prevent model collapse. Increasing the training dataset size fails to mitigate model collapse, as evidenced by the significant drop in Recall and Precision metrics. In contrast, our approach, which fine-tunes only localized cross- attention layers, demonstrates consistent performance regardless of training set size.
Ours 20% Ours 50%
Ours 70k
Ours 150k
Ours 2003
Full model 74k
Full model 200k
OCR F1
CLIP-T
0.835
0.54
0.850
0.52
0.840
0.18
0.835
0.18
0.830
3000
6000
9000
12000
3000
0000
9000
12000
Training Step
Training Step
Figure 9: LoRA fine-tuning of localized layers outperforms fine-tuning of all cross-attention layers, even with smaller datasets. LoRA fine-tuning of localized layers achieves consistent per- formance across all evaluated training set sizes, from 20k to 200k samples. While increasing the dataset size slightly improves the performance of the model when all cross-attention layers are fine- tuned, a noticeable performance gap remains compared to localized fine-tuning.
17

---

Published as a conference paper at ICLR 2025
E STUDY ON THE NUMBER OF INJECTED LAYERS IN SDXL
We carry out the study on the number of injected layers in SDXL in Table 5 We observe that leveraging more layers for the injection implies higher alignment visual text the target prompt while lowering the background preservation to the source prompt. Using 3 layers in the Stable Diffusion XL model leads obtaining the final image with text nearly good if injected to all the layers, yet preserves the background close to 1-layer injection.
Table 5: Preservation-cdition trade-off in SD-XL. Injecting the target prompt into more layers enhances the text edition but also preserves less background from the source prompt.
# layers injected (layers idx)
Image Alignment
OCR F1
CLIP-T
MSE L
SSIM
PSNR f
Texts
Texty f
Ps
Pr
0 (-)
0.00
1.00
148.13
0.34
0.19
0.85
0.71
1 (55)
17.63
0.92
36.88
0.28
0.20
0.82
0.73
2 (55,56)
22.27
0.90
35.73
0.20
0.30
0.75
0.81
3 (55,56,57)
23.38
0.90
35.43
0.19
0.32
0.74
0.82
10 (54.55....,63)
25.02
0.89
34.97
0.19
0.33
0.74
0.82
30 (34,36....66)
53.83
0.78
31.13
0.19
0.34
0.73
0.84
70 (0,1,...669)
65.98
0.72
30.17
0.19
0.34
0.72
0.85
For the images presented in Figure we use the following prompts.
Source prompts:
('A sign saying "teatre" in the aquarium', "handball" written on autumn leaves floating on the lake', A robot is holding a sign that says "mad"', 1 The slegan "sponsors" " is written on the schoolbag')
Target prompts:
['A sign saying "sponsors" in the aquarium', r ""broadway" written on autumn leaves floating on the lake', 'A robot is holding a sign that says "wisdom"', The slogan "raimundo" is written on the schoolbag')
SAFE DIFFUSION SAFETY CONCEPTS
To reproduce the results for Safe Diffusion (Schramowski al., 2023) in Section 5.3 we first run this method to all of analyzed models on the original safety concept introduced in the Safe Diffusion work: "hate, harassment, violence, suffering, humiliation, harm, suicide, sexual, rudity, bodily fluids, blood, obscene gestures, illegal activity, drug use, theft, vandalism, weapons, child abuse, brutality, cruelty". Additionally, for a fair comparison, we also ask ChatGPT to extend this safety concept in order to prevent generation of toxic textual content hate, harassment, violence, suffering, humiliation, harm, suicide, sexual, nudity, bodily fluids, blood, obscene gestures, illegal activity, drug use, theft, vandalism, weapons, child abuse, brutality, cruelry, profanity, toxic language, slurs, derogatory remarks, demeaning or offensive language, racist, sexist, homophobic, xenophobic, otherwise harmful content aimed at individuals or groups, hate speech, infla inflammatory remarks, insults, and any form of dehumanization".
image to: "an image showing
18

---

Published as a conference paper at ICLR 2025
G ON PREVENTING TOXIC GENERATIONS WITH PROMPT SUBSTITUTION
The simplest solution for preventing toxic textual content generation with text-to-image models is automatically detecting and substituting the harmful part of the original user's prompt. However, we argue that toxic language plays an important role in communicating users' intentions, which might not be adequately expressed in the final generation when automatically replacing the original prompt. As observed by Jay & Janschewitz (2008) The main purpose of swearing is to express emotions, especially anger; and frustration so even though we might not want to directly generate samples with toxic text, we still want them to align with user's emotions, and reflect them in the generation. Nevertheless, because of the rross-attention mechanism, where tokens representing text can influence other aspects of the image, this might not be possible with automatic prompt replacement. To further highlight this problem, we run a simple experiment where we measure the influence of textual content the emotional tone of the final generation. To that end, we generate samples with Stable Diffusion 3 model, following the prompt:
...A photo of a young man holding sign ..[XYZ),, visible strong emotion on his face"
where we replace [XYZ] with either toxic text, or it's clean version as suggested by ChatGPT as presented in Table
Table 6: Examples of toxic prompts with their substitutions as suggested by ChatGPT when asked to remove only toxic words with as small changes to the text as possible. (* added by authors).
Toxic text
LLM-suggested replacement
Fo*k this s#xt
Forget this mess
Go fook yourself, dssk
Go fix yourself. dude
Fauk off, issot
You'l re a pain
You deab fank
Back off. fool
Ss *w you je*k
S* * #w you, pal
You faoking You't re so fonked
You clumsy You're 'e so doomed
We sample 10 generations for each prompt and their replacement suggestions, with the same ran- dom seeds pairwise. Given set of toxic and non-toxic generations, compare the expression read from the face of a man holding toxic or non-toxic sign. To that end, we use the DeepFace li- brary (Serengil & Ozpinar 202T) that implements methods for detecting seven basic emotions from facial expressions. In Figure 10 we present the results of this comparison. As visible, substituting toxic words with benign ones on the sign significantly affects the expression on the face of the man holding that sign. This can be especially visible with the reduced score for angry and higher score for neutral expressions. At the same time, substituting text with our method does not reduce the emotional tone of the generation observed through the facial expression of the generated individ- ual. We can observe no increase in the score for neutral expression, while for some examples, the angry expression has changed more towards fear, which shares similar features. We present several generations from this experiment in Figure|
19

---

Published as a conference paper at ICLR 2025
Prompt type
Toxic
Ours
Non-toxic
40
Score
30
20
10
angry
disgust
fear
happy
sad
surprise
neutral
Figure 10: Comparison of facial expression scores (average), extracted from generations of a man holding sign with toxic texts. We compare original generations from Stable Diffusion 3 (blue), our method (orange), where we substitute the prompt only in the selected layer of the SD3, and prompt swap (green), where substitute the prompt with the LLM-suggested benign for the whole model. When generating samples with the prompt changed for the whole model, we can observe a drop in scores for the angry and fear emotions in favor of increased neutral facial expression.
Toxic
this C
prompt
You're
YOU.
OPF
Ours
You're pai!
you pal.
Back off.,
Prompt swap
Parget thamess
Tw you
You're
Back off., fool..
pal.
Figure 11: Influence of generated text on the final generation. From top: original generation with toxic text from Stable Diffusion 3, middle: generation using method (where the LLM suggested rephrasing is applied only to the one layer of the SD3 model), and bottom: generation with a prompt swap (when the suggested altered prompt is applied all layers of the diffusion model). Our method is able to generate images without toxic textual content while not affecting the emo- tional tone of the remaining part of the generation.
H TOXIC TEXT PREVENTION EXAMPLES
In Figure 12 we show, for the Deepfloyd IF model, the qualitative comparisons of our method to Negative Prompt, Safe Diffusion, and Prompt Swap.
20

---

Published as a conference paper at ICLR 2025
Linits
ALGCES L LiMiTS
OFFICAL CFFFICAL
G G HOLE
HOLE
HOLE
OFICIAL
US-US
LIS
US
SWILZERLANDS
Swittzzlerands
US
8
!!
BRANGE
B
B
RANGE
Original Image
Ours
Negative Prompt
Safe Diffusion
Prompt Swap
Figure 12:
Example results for methods for preventing toxic text in generated images. Negative Prompt and Safe Diffusion methods are incapable of removing foul words from the images. In Prompt Swap, the background of generated images is highly influenced by the suggested word. We show that our method successfully changes foul words yet ensures minimal changes to the other visual aspects of the image. Orange bounding boxes were added by the authors to cover four words.
I PSEUDOCODE FOR LAYER LOCALIZATION
We present in Algorithm our method for creating a subset of diffusion model layers that control the content of visual text generated on images.
Algorithm Finding subset of layers Lours responsible for textual content generation
Require: Ps: set of source prompts, Pr: set of target prompts, L: of indices of cross-/joint- attention layers, 0: threshold for acceptable OCR Fi-Score difference
Ensure: Lours:: of selected cross-attention layers
LF, + []
>> initialize list of mean Fı-Scores for layers
Lours < 0
N < PPsl
for 1 do
compute -Scorss for each layer via patching I..N +- images generated with LA (2) receiving Ps and receiving Pr
TI..N + text detected in II.NN using an OCR model
SI.NN + F, Score between Ti.., and Pr
LF, [1] t N Σ, EI..N
end for
Imaz + arg max, T.
Trours {mmaz}
for 1 E L {lmaa) do
I> create a set of text control layers
(LF, [maz)]
LF [1]) 0 then Lours t Lours U (1}
end if
end for return: Leurs
21

---

Published as a conference paper at ICLR 2025
J PARAMETER LOCALIZATION FOR THE TEXT STYLE
In this section, we examine whether the cross-attention layers we localize in Section 4.2 control not only the content of the visual text generated in the images but also its style.
Experiment setup. We the Stable Diffusion 3 model, which, of all those tested, exhibits the best accuracy in generating text with the style specified in the prompt. We target four text styles: handwritten, neon, graffiti and comic. In this setup, both our source prompts ps and target prompts Pr contain the same textual content to be generated but differ in the style of the text. In our exper- iments, we generate four sentences with the diffusion model: 'hello world!', 'happy new year', love you', and 'Welcome to Asia'. To ensure generalization and make sure that we do not localize layers for individual prompts, we use four prompt templates:
[' Road sign with (style) text saying (sentence)', Notebook page with a (style) text saying (sentence)', Street wall covered in (style) text saying (sentence)', Bus stop advertisement with (style) text saying [sentence)', Urban skatepark zamp with (style) text saying (sentence)"]
For measuring how particular layer ( controls the style of the text. we perform the patching tech- nique in the same way as described in Section 4.1 and calculate CLIP-T alignment between the generated images (after patching the keys and values in joint- layer /) and texts 'text in style' where s is a style from a target prompt pr-
Results. In Figure we show that the layer we localize in Section 4.2 for controlling content in visual text generated does not control the style of the text (left). Furthermore, show (right) that the Stable Diffusion 3 model, there no single layer indicating the style of the generated text and that control over style in this model is distributed over multiple layers. To support this claim, we perform a study where we iteratively add the next layers with the highest response in the previous experiment and check how many of them needed for the style to be modified. As shown in Figure 14. it is necessary to patch at least 7 out of 24 layers to change the style of the generated text. However, the images resulting from patching terms of other visual aspects. This shows that there is no layer-based separation of text style from the rest of the image elements in the Stable Diffusion model, which makes our observations regarding textual content even more unique.
many layers also significantly different in
will
HAPPY NEW
WELCOME
ASA
love youu You
0.532
hello world
Welemm To Asia
LAME YOU
0.550
23
Gesaries was Morlt
lelte wHlLd!
WELCOMME ST MGIA
i love you
Index of join attention layer
NEYDAR
Figure 13: The text style is not controlled by the same layer as the textual content. We show example generations (left) indicating that the layer we localize for determining the content of the text in generated images is not capable of changing the style of the text in images. Also, we show (right) that control over the style of the text is distributed multiple cross-attention layers in SD3 by plotting and calculating CLIP-T alignment between generations after patching particular layers with the desired text style.
22

---

Published as a conference paper at ICLR 2025
hondertton lerra
kedlo wold!
Lals wi!!!!
Walftst handertiton
Weleomee ASAA
Welcome To ASiA
Welcome To ASiA.
Welcome Asia
Welcome to Asia
aeoo comic
ASIA
ASIA
ASIA
layer
2 layers patched
4 layers patched
layers
layers
patched
patched
patched
Figure 14: The style of the text in Stable Diffusion 3 is influenced by at least 7 layers. We provide results demonstrating performance editing textual style when patching an increasing number of layers in the diffusion model. Although modifying this feature becomes feasible with 7 layers, it significantly alters the image background as well.
K
IMAGE EDITION ON LONGER TEXTS
In the Figure 15 we include examples of text editing realized using our method for DeepFloyd IF (a) and Stable Diffusion (b) models. Presented generations indicate that our localization technique can be used to edit images with longer visual text. Some examples contain errors like omitted letters or words. We believe that our performance text-based image editing strongly relies on the quality of the text generated by the diffusion model.
Additionally, we present text and image alignment metrics for image edition with our approach for varying number of words in the prompt in Table 7
Table 7: Performance metrics of SD3 image edition for varying number of words.
# words
MSE
SSIM
PSNR
OCR F1
CLIP-T 1
0.677
0.695
0.302
0.377
0.746
2
0.706
0.675
0.300
0.403
0.717
3
0.703
0.676
0.300
0.442
0.721
4
0.725
0.668
0.298
0.457
0.714
5
0.726
0.664
0.298
0.474
0.698
6
0.718
0.663
0.299
0.487
0.701
7
0.724
0.654
0.298
0.489
0.704
8
0.735
0.653
0.297
0.494
0.695
23

---

Published as a conference paper at ICLR 2025
ifussion Models The Best
Source prompt generation
ATTENTIION ALL YOU NEED
NOT ALOWND
2024
HOF PATER
ctiffied Flows the Best
Text edition result
LOCAMIATTIO
NOT ACORED
2025
ALL YOÚ NEED
(a) DecpFloyd IF
Me'ry
prompt generation
Christras, my deai!
sruut reddlls rock!
Clax Mask
GIV POR
Happy Birrthday, my dear!
edition result
teminp madels roch!
GVE MORE
(b) Stable Diffusion 3
Figure 15: Example results from editing synthetic images by leveraging parameter localization. Presented generations show that the edition can be performed for images with varying lengths of text. We show generations for models capable of generating longer visual texts: DeepFloyd IF (a) and Stable Diffusion 3 (b).
24


Reviews:
[{"review_id": "4UMAVlngHU", "reviewer": "Reviewer_RLCH", "summary": "This paper discovered that only a small amount of parameters within modern diffusion models are in charge of textual content generation. This phenomenon is widely observed among U-Net-based diffusion models like SDXL and transformer-based ones like SD3. Based on this observation, this paper developed three possible application scenarios, including LoRA-based fine-tuning for enhanced text generation, textual content editing for images generated by diffusion models, and the prevention of toxic text generation.", "questions": "Request the authors clarify the necessity of using their approach to prevent toxic textual content generation.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- By careful experimental analysis, this paper localized a small subset of cross and joint attention layers in diffusion models that are responsible for textual content generation.\n- Based on this observation, this paper developed an effective fine-tuning strategy for enhanced text generation while maintaining the model’s overall generation performance.", "weaknesses": "For the application of preventing the generation of toxic text within images, this paper claimed that their approach is able to remove the toxic text in the text prompt from the generated images. They achieved this target by first detecting the toxic text in the text prompt using state-of-the-art large language models, and then replacing the toxic text with non-harmful text for image generation using their approach. However, it seems that directly replacing the toxic text with non-harmful text and feeding the modified text prompt to the diffusion models can achieve the same target. Since the image generated by the source prompt won’t be provided to the users, there seems no need to prevent other visual content from being influenced when changing the source prompt (with toxic text) to the target prompt (without toxic text).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOdTBIVGhb", "reviewer": "Reviewer_TSpC", "summary": "This paper proposes to localize the attention layers which influence the text-rendering ability of tex-to-image generation models. With the localized layers, the authors propose to utilize them in many tasks including generation and editing.", "questions": "Since MARIO-10M dataset is utilized in training, why is MARIO-Eval Benchmark not directly used to evaluate the generation performance in the paper? \n\nWhat is the reason that the authors only use a small subset of Mario-10M dataset?\n\nHow is the editing performance compared to other methods like textDiffuser?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Distribution of image samples with text may be quite different from image samples without text. As a result, we may face a trade-off between image quality and text-rendering ability in practice when training text-to-image generation model.\nThus the idea of localizing corresponding layers for text rendering ability in text-to-image generation models is interesting. \nIf we can localize such layers, then we can use carefully designed fine-tuning so that the resulting model performs well in both image quality and text accuracy.\n\n\n\nFrom the experiments, we can see that the localized parameters indeed influence the text-rendering ability.", "weaknesses": "The fine-tuning experiment is conducted on a small subset of MARIO-10M dataset. So it is expected that fine-tuning the whole model may lead to overfitting. The experiment results can show fine-tuning localized layers indeed works, but it can not show that fine-tuning only localized layers is better than fine-tuning the whole model.\nTo illustrate the effects of the localized layers, it is suggested to conduct the experiment on large-scale dataset, i.e. fine-tuning the localized layers or whole model on all samples from MARIO-10M dataset.\n\nAnother case which is more related to the fine-tuning experiment with smaller dataset is fine-tuning the model on complicated samples of high-quality. Different from MARIO-10M which may contain some simple and low-quality samples, high-quality samples with long sentences rendered might be difficult to collect. It can strength the contribution of the proposed method if it can improve model performance under this setting. \n\n\nThe author mentioned \"We focus on LoRA for SDXL since this model has a significantly lower text generation\nquality compared to other studied DMs\". Can fine-tuned SDXL outperform SOTA methods on generation task with commonly used benchmark? If it can not, why not fine-tuning one of the SOTA models to show that the proposed method also works and can be used to improve not only SDXL? Only fine-tuning a model with lower performance is not convincing.\n\nMore qualitative results (especially with more words to be generated) and human evaluation are suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCJTD62N3j", "reviewer": "Reviewer_aXtp", "summary": "This paper made an interesting observation on how visual text specified for text-to-image generation is realized through the cross-attention layers - there are only a few cross-attention layers responsible for processing the information about the text information. This is identified by swapping the key and values to be generated across every layer. With such observation, the authors perform several experiments showing the effectiveness of using these layers for different visual text applications.", "questions": "Minor questions and suggestions:\n\n1. Is there any specific reason to classify the target text into different buckets and use the most frequent texts as the validation set? It seems other buckets are not used.\n\n2. Lines 144 and 145 look like new paragraph names.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "[Originality]\n\nThe finding about the function of cross-attention layers in generating visual text is quite novel and interesting.\n\n[Quality]\n\nThe authors did rigorous experiments to show that only a few layers are responsible for generating visual texts with multiple experiment setups. Additionally, the authors show that these layers only focus on these visual texts instead of generating other content described by the text prompts. The applications of improving text generation, editing, and preventing the generation of toxic text also demonstrate the usefulness of the findings. \n\n\n[Clarity]\n\nThe paper did a very good job of presenting the analysis and applications.\n\n[Significance]\n\nI personally find that this is an intriguing finding about the interpretation of the text-to-image diffusion model. The paper paves a way towards understanding how text embeddings are used through attention layers to control the text-to-image model. The finding also shows promise for several downstream tasks.", "weaknesses": "I did not find significant weaknesses in the paper. However, the authors could consider the following feedback to improve the readiness and clarity of the paper:\n\n1. Although I'm fairly familiar with the architectures of SD3 and SDXL, I still need to guess how localizing by patching is different from injection. So, for SD3, are only the keys and values corresponding to text embeddings swapped by the target prompt caching? \n\n2. There are many papers in text-to-image generation/editing working on cross-attention layers and manipulating the denoising steps through keys and values. The authors should consider doing more comprehensive survey and here are some of them:\n\n[1] Kumari, Nupur, et al. \"Multi-concept customization of text-to-image diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Geyer, Michal, et al. \"TokenFlow: Consistent Diffusion Features for Consistent Video Editing.\" The Twelfth International Conference on Learning Representations.\n\n[3] Patashnik, Or, et al. \"Localizing object-level shape variations with text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[4] Phung, Quynr, et al. \"Grounded text-to-image synthesis with attention refocusing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] Tumanyan, Narek, et al. \"Plug-and-play diffusion features for text-driven image-to-image translation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KOz8HiMuOJ", "reviewer": "Reviewer_9sEa", "summary": "This paper explores to find a small subset of attention layers in DMs which determines text generated in images. It finds only a few layers related to this. Based on this observation, this paper further introduces a new fine-tuning strategy and a text edit technique. It can be used to prevent the generation of harmful or toxic text in images. Extensive experiments and analysis are conducted to demonstrate the observation and effectiveness of the proposed method.", "questions": "Hope the author can answer the questions in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured, with a logical progression from the introduction to the conclusions. The writing is clear and concise, allowing complex exploration to be conveyed effectively. Key terms are introduced and defined appropriately, enhancing readability and ensuring that readers can follow the arguments easily. This observation is interesting and can effectively help us understand the inherent properties of the DMs in textual generation.", "weaknesses": "1、\tThe generalizability of the observations proposed in the paper needs further exploration. For instance, it remains to be determined whether this property holds consistently across texts of varying lengths and images with different aspect ratios.\n\n2、\tDifferent attention layers tend to focus on varying aspects, such as content or style. This has already been explored in previous works. This paper applies this feature to textual generation, yet more clarification is needed to highlight the novelty of this contribution.\n\n3、\tFrom some visualizations, it appears that the proposed approach not only alters the content of the text but also changes its style, such as the font. Are both style and content localized within the same layer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZyCuDLVnZn", "reviewer": "Reviewer_mH7c", "summary": "The paper introduces a method for precisely localizing the parameters within diffusion models (DMs) responsible for generating textual content in images. The key findings reveal that less than 1% of parameters, specifically in cross-attention and joint-attention layers, influence text generation. This localization approach, applicable across various DM architectures, including U-Net and transformer-based models, enables several enhancements:\n1. Efficient finetuning: By targeting only the localized attention layers, fine-tuning improves text generation capabilities while preserving image quality and diversity.\n2. Text editing: The method allows for precise text editing within generated images without altering other visual attributes.\n3. Toxic content prevention: The localized layer approach provides a cost-free way to prevent generating harmful text in images.\nThe authors validate their method across different models, demonstrating improvements in text alignment and visual consistency while showcasing its adaptability and efficiency across architectures.", "questions": "Could the authors provide more details on the method for identifying the critical cross and joint attention layers? Specifically, is the layer selection based solely on empirical observations, or are there theoretical underpinnings guiding this choice?\n\nHave the authors tested whether fine-tuning only the localized layers with LoRA introduces any long-term risks, such as overfitting or loss of generalization? Are there any benchmarks or datasets where this approach might underperform relative to fine-tuning all cross-attention layers?\n\nHow does the method handle diffusion models that utilize text encoders other than CLIP and T5, or models with proprietary modifications? Does the localization process adapt effectively to these scenarios, or are there specific challenges?\n\nWhy were SimpleBench and CreativeBench chosen as the primary benchmarks? Would the method be equally effective on more complex, real-world benchmarks that include diverse text styles, such as cursive or stylized fonts?\n\nCould the approach for localizing parameters influencing text generation extend to other applications? The paper would benefit from a discussion of possible applications beyond text generation and toxicity prevention, potentially sparking ideas for future work or new applications of the method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is innovative in isolating specific cross and joint attention layers that directly influence text generation in diffusion models. This novel approach to localizing model parameters is broadly applicable across various architectures, including U-Net and transformer-based models, making it an impactful step forward in the interpretability and control of DMs. Furthermore, applying this localization technique to prevent toxic text generation is a unique and practical application, expanding the scope of ethical safeguards in generative models.\nThis research has notable significance for both practical applications and theoretical advancements in generative AI. By pinpointing and leveraging specific layers responsible for text content, the paper introduces a method that improves the efficiency and quality of DMs in text generation tasks. The application to toxic text prevention adds a meaningful dimension, contributing to the broader conversation on safe and ethical AI. The findings open up new avenues for fine-tuned, targeted adjustments in diffusion models, making it easier for practitioners to achieve high-quality text rendering and control content generation more responsibly.", "weaknesses": "The method for identifying key cross and joint attention layers is central to the paper’s contribution, yet the description of this process could benefit from greater specificity. For instance, while the authors mention leveraging activation patching techniques, the exact steps involved in isolating specific layers and evaluating their effect on text generation are not fully detailed. Providing pseudocode or a workflow diagram would make the process clearer and more reproducible.\nThe approach is tested on models using CLIP and T5 encoders, yet diffusion models incorporating different or proprietary text encoders could present challenges to the proposed localization technique. Extending the evaluations to other popular encoders (or discussing limitations for certain encoder types) would improve the generalizability of the method.\nWhile LoRA fine-tuning on localized layers is shown to enhance text generation quality, the paper does not deeply analyze potential trade-offs involved. Specifically, fine-tuning localized layers only may limit generalization to unseen prompts or introduce subtle biases in text rendering quality across different styles or languages. Including a comparative analysis of localized vs. whole-model fine-tuning would be helpful, especially with examples illustrating where one approach might be preferred over the other.\nWhile the method is shown to be effective on certain models, scaling the approach for larger models or more computationally intensive tasks may present challenges. A more detailed discussion on these potential limitations would provide a balanced view and help guide future research in optimizing model efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)