Localizing and Editing Knowledge In Text-to-Image Generative Models
Authors: Samyadeep Basu
Nanxuan Zhao
Vlad I Morariu
Soheil Feizi
Varun Manjunatha
Venue: ICLR 2024 poster (2024)
Keywords: text-to-image
interpretability
model editing
Primary area: generative models
TL;DR: We investigate how knowledge is stored in text-to-image models and use it to design a fast model editing method

Abstract: Text-to-Image Diffusion Models such as Stable-Diffusion and Imagen have achieved unprecedented quality of photorealism with state-of-the-art FID scores on MS-COCO and other generation benchmarks. Given a caption, image generation requires fine-grained knowledge about attributes such as object structure, style, and viewpoint amongst others. Where does this information reside in text-to-image generative models? In our paper, we tackle this question and understand how knowledge corresponding to distinct visual attributes is stored in large-scale text-to-image diffusion models. We adapt Causal Mediation Analysis for text-to-image models and trace knowledge about distinct visual attributes to various (causal) components in the (i) UNet and (ii) text-encoder of the diffusion model. 
In particular, we show that unlike large-language models, knowledge about different attributes is not localized in isolated components, but is instead distributed amongst a set of components in the conditional UNet. These sets of components are often distinct for different visual attributes (e.g.,  style} / objects).  Remarkably, we find that the text-encoder in public text-to-image models such as Stable-Diffusion contains {\it only} one causal state across different visual attributes, and this is the first self-attention layer corresponding to the last subject token of the attribute in the caption. This is in stark contrast to the causal states in other language models which are often the mid-MLP layers.  Based on this observation of only one causal state in the text-encoder, we introduce a fast, data-free model editing method DiffQuickFix which can effectively edit concepts (remove or update knowledge) in text-to-image models. DiffQuickFix can edit (ablate) concepts in under a second with a closed-form update, providing a significant 1000x speedup and comparable editing performance to existing fine-tuning based editing methods.

Preprint
LOCALIZING AND EDITING KNOWLEDGE IN TEXT-TO-IMAGE GENERATIVE MODELS
Samyadeep Nanxuan Zhao?, Vlad Morariu Soheil Feizi*!, Varun Manjunatha University of Maryland, 2: Adobe Research;
Correspendence to: t0:sbasul2 umd. edu
ABSTRACT
Text-to-Image Diffusion Models such as Stable-Diffusior and Imagen have achieved unprecedented quality of photorealism with state-of-the-art FID scores on MS-COCO and other generation benchmarks. Given caption, image gen- eration requires fine-grained knowledge about attributes such object structure, style, and viewpoint amongst others. Where does this information reside in text- to-image generative models? In our paper, we tackle this question and understand how knowledge corresponding to distinct visual attributes is stored in large-scale text-to-image diffusion models. We adapt Causal Mediation Analysis for text- to-image models and trace knowledge about distinct visual attributes to various (causal) components in the (i) UNet and (ii) text-encoder of the diffusion model. In particular, we show that unlike large-language models, knowledge about differ- ent attributes is not localized in isolated components, but instead distributed amongst a set of components in the conditional UNet. These sets of compo- nents are often distinct for different visual attributes (e.g., style objects). Re- markably, we find that the text-encoder in public text-to-image models such Stable-Diffusion contains only one causal state across different visual attributes, and this is the first self-attention layer corresponding to the last subject token of the attribute in the caption. This is in stark contrast to the causal states in other language models which are often the mid-MLP layers. Based on this observa- tion of only one causal state in the text-encoder, we introduce a fast, data-free model editing method DIFF- QUICKFIX which can effectively edit concepts (re- move or update knowledge) in text-to-image models. DUr--UICKFIX can edit (ablate) concepts in under a second with a closed-form update, providing a signif- icant 1000x speedup and comparable editing performance to existing fine-tuning based editing methods. Code at
INTRODUCTION
Text-to- Image generative moce as Stable-Diffusine (Rombach et al. (202)), Imagen (Saharia al. 2022) and DALLE (Ramesh et al. 202T) have revolutionized conditional image generation in the last few years. These models have attracted lot of attention due to their impressive im- age generation and editing capabilities, obtaining sate--of-thh-rt FID scores on common generation benchmarks such MS-COCO (Lin et al., 2014). Text-to-Image generation models are gener- ally trained on billion-scale image-text pairs such as LAION-5B (Schuhmann et al., 2022) which typically consist of plethora of visual concepts encompassing color, artistic styles, objects, and famous personalities, amongst others. Prior works (Carlini et al. 2023 Somepalli et al. 2023a b) have shown that text-to-image models such Stable-Diffusion memorize various aspects of the pre-training dataset. For example, given caption from the LAION dataset, model can generate an exact image from the training dataset corresponding to the caption in certain cases (Carlini al.) 2023). These observations reinforce that some form of knowledge corresponding to visual attributes stored in the parameter space of text-to-image model.
When image is generated, it possesses visual attributes such as (but not limited to) the presence of distinct objects with their own characteristies (such as color or texture), artistic style or scene
Equal Advising

---

Preprint
Comppted State)
Centrrteed Image intittuulss to caption
Teal treektii Aeple Tervig
[4]
[c]
Redoration on Carnupted Model Soatel
Computtd Model
Genelated image
Ten-drodder
Apple Gogh
Ter breeder
Myple in laan Gogh Style
castion
(b)
cormuziondng theth scribing inseett
Serl-Attn
Resheet
Text Encoder Copy
model
Cormupted States
Figure I: Causal Tracing in Text-to-Image Models for (i) UNet and (ii) Text-Encoder shows that knowledge location matters, i.e., restoring causal layers in corrupted model causes the model to obey the prompt again, while restoring non-causal layers does not. (a) Clean Model: We prompt Stable-Diffusiie model in the conventional way and generate an image as output. (b) Corrupted Model: Token embeddings corresponding to attribute of interest are corrupted, leading to generated image that does not obey the prompt. (c) Restored (Causal) Model: Causal layer acti- vations are now copied from the clean model to the corrupted model. We observe that the corrupted model can now generate images with high fidelity to the original caption. (d) Restored (Nnn-Cusal)) Model: Non-causal layer activations are copied from the clean model to the corrupted model, but we now observe that the generated image does not obey the prompt. A single layer is copied time, and it can be from either the UNet (solid violet arrow) or the text-encoder (broken black arrow).
viewpoint. This attribute-specific information is usually specified in the conditioning textual prompt to the UNet in text-to-image models which is used to pull relevant knowledge from the UNet to construct and subsequently generate an image. This leads to an important question: How and where is knowledge corresponding to various visual astributes stored in text-to- tot-toi-iage models? In our paper, we choose concentrate on specific visual attributes within a scene, including objects, style, action, and color. Additionally, in the Appendix, we delve into aspects like count and viewpoint.
In this work, we empirically study this question towards understanding how knowledge correspond- ing to different visual attributes is stored in text-to-image models, using Stable Inffusion(Rombat
ct al. 2021 as representative model. In particular, we adapt Causal Mediation Analysis (Vig et al.) 2020 Pearl 2001) for large-scale text-to-image diffusion models to identify specific causal compo- nents in the (i) UNet and (ii) the text-encoder where visual attribute knowledge resides. Previously, Causal Meditation Analysis has been used for understanding where factual knowledge is stored in LLMs. In particular, (Meng et al. 2022) find that factual knowledge is localized and stored in the mid-MLP layers of I.M. such GPT-J (Wang & Komatsuzaki (02T). Our work, however, paints a different picture- for multimodal text-to-image models, we specifically find that knowledge is not localized one particular component. Instead, there exist various components in the UNet where knowledge is stored. However, each of these components store attribute information with a different efficacy and often different attributes have a distinct set of causal components where knowledge is stored. For e.g., for style we find that the first self-attention layer in the UNet stores style related knowledge, however is not causally important for other attributes such as objects, viewpoint or action. To our surprise, we specifically find that the cross-attention layers are not causally important states and a significant amount of knowledge is in fact stored in components such as the ResNet blocks and the self-ateention blocks.
Remarkably, in the text-encoder, we find that knowledge corresponding to distinct attributes is strongly localized, contrary to the UNet. However unlike generative language models (Meng al.) 2022) where the mid MLP layers are causal states, we find that the first self-attention layer is causal in the CLIP based text-encoder of public text-to-image generative models (e.g., Stable Daffu--ifffuion).
Identification of local causal states in given model has a crucial benefit: it allows for incorporating controlled edits the model by updating only tiny fraction of the model parameters without any
2

---

Preprint
fine-tuning. Using our observation that the text-encoder hosts only localized causal state, introduce a new tata-free and fast model editing method DIFF-QUICKFIX which can edit concepts in lext-to-image models effectively using closed-form update. In particular, show that DIFF- QUICKFIX can (i) remove copyrighted styles, (ii) trademarked objects as well as (iii) update stale knowledge 1000x faster than existing fine-tuning based editing methods such (Kumari et al.) 2023 Gandikota et al. 2023) with comparable or even better performance in some cases.
In summary, our contributions are as follows:
We adapt Causal Mediation Analysis (Pearl 2001; Vig et al. 2020) to large-scale text-to- image models (with Stable-Diffusion as representative model), and it to trace knowl- edge corresponding to various visual attributes in the UNet and text-encoder.
We perform large-scale analysis of the identified causal components and shed light on the knowledge flow corresponding various visual attributes in the UNet and the text-encoder.
Leveraging the interpretability observations of localized causal states in the text-encoder, we develop a light-weight method DIrF-QUICKFIX which can edit various concepts in text-to-image models in under second, 1000x faster than existing concept ablating meth- ods Kumari et al. (203);; Gandikota et al. 2023)
2 RELATED WORKS
Tmmage Diffusion Models. In the last year, large number of text-to-image models such as Stable-Diffusion (Rombach al. 202T), DALLE (Ramesh et al., (021) Imagen (Saharia al.]
2022) and others (Balaji et al. 2022 Chang al. 2023 Ding et al. 2022 Kang et 2023) have been released. addition, the open-source community has released DeepFloyd and Midjourney which can generate photorealistic images given text prompt. While most of these models operate in the latent space of the images, they differ in the text-encoder used. For e.g., Stable-Diffusion uses CLIP for the text-encoder, whereas Imagen uses T5. These rext-to-mmaee diffusion models have been used as a basis for various applications such as image-editing, semantic-segmentation, object-detection, image restoration and zero-shot classification.
Intepretability of Text-to-Image Models. To our knowledge, few works delve into the mech- anisms of large text-to-image models like Stable-Diffusion. DAAM (Tang al.) (023) interprets diffusion models by analyzing cross-attention maps between text tokens and images, emphasizing their semantic accuracy for interpretation. In contrast, our approach focuses on comprehending the inner workings of diffusion models by investigating the storage of visual knowledge related to different attributes. We explore various model layers beyond just the cross-attention layer.
Editing Text-to-Image Models. Understanding knowledge storage in diffusion models has signif- icant implications for model editing. This ability to modify a diffusion model's behavior without retraining from scratch were first explored in Concept- Abbation (Kumari et al. 2023) and Concept- Erasure (Gandikota et al., 2023). TIME (Orgad et al. 2023) is another model editing method which translates between concepts by modifying the key and value matrices in cross-attention layers. How- ever, the experiments in (Orgad et al. (203)) do not specifically target removing or updating concepts such those used in (Kumari al. 2023 Gandikota et al.) 2023). We also acknowledge concurrent works (Gandikota et al. 2024) and (Anonymous) 2023) use a closed-form update on the cross- attention layers and text-encoder respectively abl concepts. However,
note that work focuses primarily on first understanding how knowledge is stored in text-to-image models and sub- sequently using this information to design a closed-form editing method for editing concepts.
3 CAUSAL TRACING FOR TEXT-TO-IMAGE GENERATIVE MODELS
In this section, we first provide a brief overview of diffusion models in Sec. Sec.(31| 33. I. We then describe how causal tracing is adapted multimodal diffusion models such Stable-Diffusion
'htps://www.deepflyldaii hhtps:/wwwwmiiijuumy.comm
3

---

Preprint
3.1 BACKGROUND
Diffusion models are inspired by non-equilibrium thermodynamics and specifically aim to learn to denoise data through number of steps. Usually, noise is added the data following Markov chain across multiple time-steps & E [0. T). Starting from an initial random real image Xg, the noisy image at time-step t is defined as = /a,Xo + v(1 os)c. In particular, strength of the random Gaussian noise and it gradually decreases as the time-step increases such that XT ~ (0, I). The denoising network denoted by ex(x;, C, t) is pre-trained to denoise the noisy image X to obtain X Usually, the conditional input с to the denoising network eø (.) is a text-embedding of a caption through text-encoder By which is paired with the original real image Xg. The pre- training objective for diffusion models can be defined as follows for a given image-text pair denoted by (x, c):
determines the
[(x, c) = Ee,l//c- co c, c.2)
(1)
where 9 is the set of learnable parameters. For better training efficiency. the noising as well as the denoising operation occurs in a latent space defined by z = E(x) Rombach et al. (2021). In this case, the pre-training objective learns denoise in the latent space denoted by:
L(x, c) = E.,.Icc - co c0(2,... c, 1),
(2)
where 2, E(x) and & is an encoder such VQ-VAE (van den Oord et al. 2017). During inference, where the objective is to synthesize an image given a text-condition c, a random Gaussian noise XT (N, is iteratively denoised for fixed range of time-steps in order to produce the final image. We provide more details on the pre-training and inference steps in Appendix
3.2 ADAPTING CAUSAL TRACING FOR TEXT-TO-IMAGE DIFFUSION MODELS
Causal Mediation Analysis PPeal 2001: Vig et al. 2020 is method from causal inference that studies the change in response variable following an intervention on intermediate variables of interest (mediators). One can think of the internal model components (e.g., specific neurons or layer activations) as mediators along a directed acyclic graph between the input and output. For text- to-image diffusion models, we use Causal Mediation Analysis to trace the causal effects of these internal model components within the UNet and the text-encoder which contributes towards the generation of images with specific visual attributes (e.g., objects, style). For example, we find the subset of model components in the text-to-image model which are causal for generating images with specific objects, styles, viewpoints, action or color.
Where is Causal Tracing Performed? We identify the causal model components in both the UNet eg and the text-encoder ey. For ea, we perform the causal tracing at the granularity of layers, whereas for the text-encoder, causal tracing is performed at the granularity of hidden states of the token embeddings in c across distinct layers. The UNet Eg consists of 70 unique layers distributed amongst (ii) mid-block and (iii) up up-block. Each of these blocks contain varying number of cross-attention layers, self-attention layers and residual layers. Fig [ visualizes the internal states of the UNet and how causal tracing for knowledge attribution is performed. For the text-encoder V-y, there are 12 blocks in total with each block consisting of self-attention layer and MLP layer (see Fig Th. We highlight that the text-encoder in text-to-image models such Stable-Diffusion has a GPT-style architecture with causal self- attentitnnion though it's pre-trained without a language modeling objective. More details on the layers used in Appendix J
three types of blocks: (i) down-
Given caption c, an image is generated starting from some random Gaussian noise. This image x encapsulates the visual properties embedded in the caption c. For e.g.. the caption e can contain information corresponding from objects to action etc. We specifically identify distinct components in the UNet and the text-encoder which are causally responsible for these properties.
Creating the Probe Captions. We primarily focus on four different visual attributes for causal tracing: (i) objects; (ii) style; (iii) color; and (iv) action. In particular, identifying the location of knowledge storage for objects and style can be useful to perform post-hoe editing of diffusion models to edit concepts (c.g., delete or update certain concepts), We provide the complete details about the probe dataset used for causal tracing Appendix A The probe dataset also contains additional captions for viewpoint and count attribute. However, we do not focus on them as often the generations from the unedited model are erroneous for these attributes (see Appendix forr details).
4

---

Preprint
Style
Original
Corrunted
Caual State Causal State
Auglone
Objects
CHI--Srore
Action
Pronpe pabr of dingg hurriss
WAR
Oyer In
bay
down-block
mid-block!
up-Block
Figure 2: Causal Tracing Results for the UNet: Knowledge is Distributed. The intensity of the bars indicate the original caption. For each attribute, we find that the causal states are distributed across the UNet and the distribution varies amongst distinct attributes. For e.g., self-attn in the first layer is causal for style, but not for objects, action or color. Similarly, mid-block cross-attn is causal for action, but not for the other attributes. On the right-side, we visualize the images generated by Original model: (ii) Corrupted Model; (iii) Restored causal states and (iv) Restored non-causal states in the UNet for style, action, object, color attributes.
between the generated image (after causal intervention) and the
3.3 TRACING KNOWLEDGE IN UNET
During inference, classifier-free guidance (Ho & Salimans, 2021) is used to regulate image- generation by incorporating scores from the conditional and unconditional diffusion model at each of the time-stes. In particular, at each time-step, classifier-free guidance used in the following way to combine the conditional (Ze, f)) and unconditional score estimates (ea (Z,, t)) at each time-step t to obtain the combined score denoted as E(Z), t): éo (ze, c, t) = co (2. c, t) + acco(Zz, c, co(ze,t)), t)), Vt E [T.
co(2,.
(3)
This combined score is used to update the latent z₁ using DDIM sampling (Song et al. 2020) at each time-step iteratively to obtain the final latent code Zp-
To perform causal tracing on the UNet eg (see Fig for visualization), we perform a sequence of operations that is somewhat analogous earlier work from (Meng al. 2022) which investigated mhowledge-tracimg in large language models. We consider three types of model configurations: (i) clean model eg, where classifier-free guidance used as default; (ii) a corrupted model egorr, where the word embedding of the subject (c.g., Van Gogh) of a given attribute (e.g., style) corresponding to a caption is corrupted with Gaussian Noise; and, (iii) a restored model cj crestored, which similar to coomt except that one of its layers is restored from the clean model at each time-step of the classifier- <g free guidance. Given list of layers A, let a, E A denote the jth layer whose importance needs to be evaluated. Let co [a:]. < [a:] and e egstooeed(a, [a:] denote the activations of layer a; To find the importance of layer a, for particular attribute embedded in caption c, we perform the following replacement operation on the corrupted model EşorT to obtain the restored model ep restored,
(4) Next, obtain the restored model by replacing the activations of layer o, of the corrupted model with those of the clean model to get a restored layer eg egestored We run classifier free guidance to obtain the combined score estimate:
Eo restored [a.] c [a,] = c=(a.]-
prestored (tt, C, 1) c Eg
restored (Ze; c, + aa(e restoredd 24, c, -2)))(ree
restored (zt, VI E [T,1]. (5)
5

---

Preprint
Causa Tracing for the Texb-Encoder (CUP-Sccore
Original
Corrupted
Causal State Causa State
Token
Different Tokens the
Last Subject Rken
Higher CLP Score
Prompr phot of sondwncn room
Rist Raknn
10
20
Layers
Prompr: deer romninng
Sey
Figure 3: Causal Tracing in the Text-Encoder: Knowledge is Localized. In the CLIP text- encoder used for self-attention layer corresponding the last subject token. The CLIP-Score(Lett) is computed across all the four visual attributes. Visualizations (Right) further illustrate that restoring the sole causal state (self-attn-O) leads to image generation with high fidelity to the original captions.
we find the existence of only one causal state, which the first
The final latent zg obtained with the score from Equation (5) al. 2020 and passed through the VQ-VAE decoder to obtain the final image x
each time-step using DDIM (Song xrestored,
3.4 TRACING KNOWLEDGE THE TEXT-ENCODER
The text-encoder in public text-to-image models such as text-encoder Rombach et al. (021). Similar to Sec. 3.3), we define three states of the CLIP text- encoder: (i) Clean model denoted by Uyy (ii) Corrupted model Us where the word embedding of the subject in a given caption is corrupted; (iii) Restored model prestored which similar to ह except that one of its layers is copied from Ey Similar to Sec. 3.3 to find the effect of the layer a, E A, where A consists of all the layers to probe in the CLIP text-encoder:
is ELIP-VIT-L/3361
By rrestored [a:] e [a:] v. [a.),
(6)
We then use the restored text-encoder oy gresorre with classifier- free guidance to obtain the final score estimate:
Eg(Ze, t) cØ (zz, t) + alea(Ze, (',)),,,'',)))) cz t)), Vt [T.
where = uf urestored [a]](c) for a given caption c. This score estimate éø d' t) at each time-step t is used to obtain the final latent code 2o which is then used with the VQ-VAE decoder to obtain the final image X
restoredd
3.5 EXTRACTING CAUSAL STATES USING CLIP-SCORE
In this section, discuss details on how retrieve causal states using automated metrics such as CLIP-Score (Hessel et al. 2021). Let x restorrdd (a:) be the final image generated by the diffusion model after intervening xorr be the final image generated by the corrupted model. In particular, we are interested in the average indirect effect (Vig al., [020 Pearl 2001) which measures the difference between the corrupted model and the restored model. Intuitively. higher value of average indirect effect (AIE) signifies that the restored model deviates from the corrupted model. To compute the average indirect effect with respect to causal mediation analysis for text-to-image models such as Stable- Sifbll-iffuuion we use CLIP-ScOr which computes the similarity between an image embedding and caption embedding. In particular, AIE (CL.PScore (x, xfestored c) CLIPScore. soorr, c)). Given x common across all the layers, we can use CLIPSCore gestored c) as the AIE.
layer d; Xo be the image generated by the clean diffusion model and
Selecting Threshold for CLIP-Score. We observe that the difference between the CLIP-Score of generated images (after restoration) with high fidelity to the original caption and generated images (after restoration) with low fidelity to the original caption, to be small. Therefore to effectively find a reasonable cut-off point to automatically select causal states (where the generated images have high-fidee high-fidelity to the original caption), we threshold selection mechanism. In order determine the optimal threshold value for attribute. To this end, establish concise user study interface (refer Appendix D for details).
we select a small validation set of 10 prompts per

---

Preprint
Through human participation, we collect binary ratings if an image generated by restoring partic- ular layer is faithful to the original captions. We then extract the common causal states across all the prompts for given attribute and find the average (across all the prompts) CLIP-Score for each causal state. We then use the lowest average SL.P-Sccore corresponding to a causal state as the threshold, which we apply on the probe dataset in Appendix to filter the causal states scale for each attribute separately.
4 HOW IS KNOWLEDGE STORED IN TEXT-TO-LMAGE MODELS?
In this section, discuss the results of tracing knowledge across various components of the text- to-image model in details.
Tracing Results for UNet. In Fig we illustrate the distribution of causal states across different visual attributes within the UNet architecture using the CLIP-SSore metric. This metric evaluates the faithfulness of the image produced by the restored state xfeatored compared to the original cap- tion From the insights derived in Fig 2 it becomes evident that causal states are spread across diverse components of the UNet. In particular, we find that the density of the causal states are more in the up-block of the UNet when compared to the down-block or the mi mid-block. Nonethe- less, a notable distinction emerges in this distribution across distinct attributes. For instance, when examining the style attribute, the initial self-attention layer demonstrates causality, whereas this causal relationship is absent for other attributes. Similarly, in the context of the action attribute, the cross-attention layer within the mid-block exhibits causality, which contrasts with its non-causal behavior concerning other visual attributes. Fig 2 showcases the images generated by restoring both causal and non-causal layers within the UNet. A comprehensive qualitative enumeration of both causal and non-causal layers for each visual attribute is provided in Appendix B Our findings underscore the presence of information pertaining to various visual attributes in regions beyond the cross-attention layers. Importantly, we observe that the distribution of information within the UNet diverges from the patterns identified in extensive generative language models, as noted in prior re- search (Meng et al. 22022), where attribute-relater knowledge is confined to a few proximate layers. In Appendix M we provide additional causal tracing results, where we add Gaussian noise to the entire text- text-embedding. Even in such a case, certain causal states can restore the model close to its original configuration, highlighting that the conditional information can be completely bypassed certain causal states are active.
Tracing Results for Text-Encoder. In Fig we illustrate the causal states in the text-encoder for Stable-Diffusion corresponding to various visual attributes. At the text-encoder level, we find that the causal states are localized to the first self-attention layer corresponding to the last subject token across all the attributes. In fact, there exists only one causal state in the text-encoder. Qualitative visualizations in Fig 3 and Appendix Cillustrrte that the restoration of layers other than the first self-attention layer corresponding to the subject token does not lead to images with high fidelity to the original caption. Remarkably, this observation is distinct from generative language models where factual knowledge is primarily localized in the proximate mid MLP layers Meng al. (022).
General Takeaway. Causal components corresponding to various visual attributes are dis- persed (with a different distribution between distinet attributes) in the UNet, whereas there exists only one causal component in the text-encoder.
The text-encoder's s strong localization of causal states for visual attributes enables controlled knowl- edge manipulation in text-to-image models, facilitating updates or removal of concepts. However, since attribute knowledge is dispersed in the UNet, targeted editing is challenging without layer in- terference. While fine-tuning methods for UNet model editing exist (Gandikota et al. 2023] Kumaar et 2023), they lack scalability and don't 't support simultancous editing of multiple concepts. In the next section, we introduce a closed-form editing method, MIFF-QUICKFIX, leveraging our causal tracing insights to efficiently edit various concepts in text-to-image models.
5 DIFF-QUICKFIX: FAST MODEL EDITING FOR TEXT-TO-IMAGE MODELS
5.1 EDITING METHON
Recent works such as (Kumari et al. 2023 Gandikota et al. 2023) edit concepts from text-to-image diffusion models by fine-tuning the UNet. They generate training data for tine-tuning using the pre- trained diffusion model itself. While both methods are effective at editing concepts, fine-tuning the
7

---

Preprint
Editing Layer Layers
Comperioon with Other Madhods
Offecisnness of Mfttoooon Abloted Mode
Locan Naris
20
"SOL "SaSr
C Store
CUP Score
33
Diffornnt Se f-ttt Layers
"ole AbMation
Type
Lause
Ediing
(6)
Figure 4: Quantitative Analysis of DIFF-QUICKFIX, (a) Editing Causal vs. Non-Causal Layers (Averaged across Objects, Style and Facts): Lower CLIF-ScOr for causal layer indicates suc- cessful edits; (b) Efficacy of DIFF-QUCCKII when compared to other methods - Our method leads comparable CLIP Scores fine-tuning based approaches, but can edit concepts 1000x faster; (c) DIFF-QUICKFIX can be used to effectively edit multiple concepts at once, shown by comparable CLIP-Scores to the single-concept edited ones.
per sonnact
UNet can be expensive due to backpropogation of gradients through the UNet. To circumvent this issue, we design a fast, data-free model editing method leveraging our interpretability observations in Section 4 where we find that there exists only causal state (the very first self-attention layer) in the text-encoder for Stable- Difffuioifni
Our editing method DIFF-QUICKFIX can update text-to-image diffusion models in targeted way in under 1s through a closed-form update making it 1000x faster than existing fine-tuning based con- cept ablating methods such (Kumari et al. 2023 Gandikota al. 2023). The first self-attention layer in the text-encoder for Stable-Diffusion contains four updatable weight matrices: WA W Wa and W where WK, W,, W, are the projection matrices for the key, query and value embeddings respectively. Wout is the projection matrix before the output from the self-attn-0 layer after the attention operations. DIFF- QUICKFIX specifically updates this Wout matrix by collecting cap- tion pairs (C& C.) where e (key) is the original caption and c (value) is the caption to which c₁ is mapped. For e.g.. to remove the style of 'Van Gogh', we set C 'Van Gogh' and G. 'Painting'. In particular, to update W Wout, we solve the following optimization problem:
Σlw [[ vell2 + Al|Wout out Worll.. W Wout
min
where À is regularizer to not deviate significantly from the original pre-trained weights W outs N denotes the total number of caption pairs containing the last subject token embeddings the key and value. k; corresponds to the embedding of Ck. after the attention operation using W We and W. for the 8th caption pair. corresponds to the embedding of Co; after the original pre-trained weights W'... out acts on it.
One can observe that Eq. (8) has a closed-form solution due to the absence of any non-linearities. In particular, the optimal Wout can be expressed the following:
N
Wout (AW' out
)AI + Σ ki
In Section 5.3
we show qualitative as well as quantitative results using DIFF-QUICKFIX for editing various concepts in text-to-image models.
5.2 EXPERIMENTAL SETUP
We validate DIFF- QUIT-QUIXKFI by applying edits to Stable-Diffusion (Rombach al.) 1021 model and quantifying the efficacy the edit. For removing concepts such as artistic styles or objects us- ing QUFF-QUICKFIX,
the prompt dataset from (Kumari et al. (023). For updating knowledge (e.g., President of a country) in text-to-image models, we add newer prompts to the prompt dataset from (Kumari et al. 2023) and provide further details in Appendix N We compare
method with ()) Original Stable-Difii Diffuusson (ii) Editing methods from (Kumari et al.) 2023) and (Gandikotal al. 2002). To validate the effectiveness of editing methods including our DIFF-QU QUI--QUICKFIX, perform evaluation using automated metrics such as
Score. In particular, we compute the CLIP-Score between the images from the edited model and the concept corresponding to the visual attribute which is edited. A low CLIP-ScOR. therefore indicates correct edits.
8

---

Preprint
Remaving Snaopy
Removing Van Goph Style
Updating the President of US
Updating the British Monarch
Removing R2D2
Removing Monet Style
Removing Nemo
Removing Grumpy Cat
After
Edit
Sefore Edit
After toit
Before Edit
Alter Edit
Figure 5: Qualitative Examples with using DIFF- SLFF-QUICKFIX to ablate style, objects and update facts in text-toiimage models. More qualitative examples in the Appendix
5.3 EDITING RESULTS
Editing Non-causal Layers Does Not Lead to Correct Edits. We use DIFF- QIFT-QUICKIIX with the non-causal self-attention layers in the text-encoder to ablate styles, objects and update facts. In Fig (a). we compute the CLIP-Score between the generated images and the attribute from the original captions (c.g., van gogh in the case of style). In particular, we find that editing the non- causal layers does not lead to any intended model changes highlighted by the high consistently across non-causal layers (layers numbered I to 11). However, editing the sole causal layer (layer-0) leads to correct model changes, highlighted by the lower CLIP-SCore between the generated images from the edited model and the attribute from the original captions. This shows that identifying the causal states in the model particularly important to perform targeted model editing for ablating concepts. In Appendix G
we show additional qualitative visualizations.
Efficacy in Removing Styles and Objects. Fig (b) shows the average CLIP-Score of the generated images from the edited model computed with the relevant attributes from the original cap- tions. We find that the CLIP-SCore from the edited model with DIFF-QUICKFI decreases when compared to the generations from the unedited model. We also find that our editing method has com- parable CLIP-Scores to other fine-tuning based approaches such as Soncept-Erase, (Gandikotal al. 2023) and Concept-Ablation (Kumari et al. [2023), which are more computationally expen- sive. Fig 5 shows qualitative visualizations corresponding to images generated by the text-to-image model before and after the edit operations. Together, these quantitative and qualitative results show is able to effectively remove various styles and objects from an underlying text-to-image model. In Appendix [ we provide additional qualitative visualizations and in Fig [2 we show additional results showing that our editing method does not harm surrounding concepts (For e.g., removing the style of Van Gogh does not harm the style of Monet).
that DIFF-QUICKEE
Efficacy in Updating Stale Knowledge. The CLIP-ScOre Score between the generated images and a caption designating the incorrect fact (e.g., Donald Trump as the President of the US) decreases from 0.28 to 0.23 after editing with DIFF-QUICKIIX while the CLIP-ScOr (c.g., Joe Biden as the President of the US) increases from 0.22 to 0.29 after the relevant edit. This shows that the incorrect fact is updated with the correct fact in the text-to-image model. Additional qualitative visualizations are provided in aldd Appendix
with the correct fact
Multiple Edits using DIFF-QUICKFIC An important feature of DIFF- QIFF-QUICKFIX is its capa- bility to ablate multiple concepts simultaneously. In Fig [c),, our framework demonstrates the removal of up 10 distinct styles and objects at once. This multi-concept ablation results in lower CLIP-Sccr Scores compared to the original model, similar CONCLUSION
to single concept editing.
Through the lens of Causal Mediation Analysis. we present methods for understanding the storage of knowledge corresponding to visual attributes in text-to-image models. Notably, we find distinct distribution of causal states across attributes the UNet, while the text-encoder maintains a single causal state. This differs significantly from observations in language models like GPT, where factual information is concentrated in mid-MLP layers. In contrast, our analysis shows that public text- to-image models like Stable-Diffusion concentrate multiple visual attributes within the first self- attention layer of the text-encoder. Harnessing these insights, we design a fast model editing method The potency of DIFF-QU SEFF-QUIIKFFXX is manifested through its adeptness in removing artistic styles, objects, and updating outdated knowledge all accomplished data-free free and in less than second, making DIFF-QUICKEF practical asset for real- world model editing scenarios.

---

Preprint
REFERENCES
Anonymous. ReFACT: Updating text-to-image models by editing the text encoder. In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL https:// openreview wwwereviewneeee/eeeeee htt/:ooorummmm/e wia.middddddlalnneem 4Hoy] under review.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. CoRR, abs/2211.01324. 2022. doi: 10.48500/RXXV2221..0011 2211. 01324
URL http:::/aoi.orr/10..888338
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270,, Anaheim, CA, August 2023. USENIX Association. ISBN 978-1-939113337--3 URL https:/ /www www.usenix.cmm o carlini
Con www.Feeeeneee/eeeee http://uennnnnnnneteeeeeeeeeeeee http://ttyytppppetttteemmmmm
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 4055 4075. PMLR, 2023, URL httptttpps://ppteeeheettt
mlr press/v202. chang html
Ming Ding, Wendi Zheng, Wenyi Hong. and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204. 142/7, 2022.
Rohit Gandikota, Joanna Materzyúska, Jaden Kauto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúska, and David Bau. Unified concept editing in diffusion models. /EEE/CVF Winter Conference on Applications of Computer Vision, 2024.
Hongcheng Gao, Hao Zhang, Yinpeng Dong, and Zhijie Deng. Evaluating the robustness of text-to- image diffusion models against real-world attacks, CoRR, abs/2306. 03103, 2023, doi: 10.48550/ ARXIV 13103. httthhttt://aatt http://............/////////////
48550/ /r.500/axxiv..
13103
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08718, 2021. URL
https::/arre org/ abs aba//aas2220 /etss/214..00 08718
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop on Deep Generative Models and Downstream Applicttion,,, 2021. htt.htttps///ppeeeeeee
net/ net//Eormm?ie id= BAkxfYbI
Minguk Kang, Jun-Yun Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Nupur Kumari, Bingliang Zhang, Sheng-Yo Yu Wang, Eli Shechtman, Richard Zhang, and Yun Zhu. Ablating concepts in text-to-image diffusion models. In 'International Conference on Com- puter Vision (ICCV), 2023.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimicc. Microsoft COCO: common objects in context. CORR, 0bs/1405.0314 2014.
org/
1405.00 0312
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
10

---

Preprint
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to- to-tagge diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncer- tainty in Artificial Intelligence, UArO1. pp. 411-420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8821-8833 PMLR, 18-24 Jul 2021. URL. htttps::/proceediin wtw.mmmw..p..s/emmm 299raa/ma22aa htt.mma....mmmmm.mmm
m
Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandin- sky: an improved text-to-image synthesis with image prior and latent diffusion. 2023.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CoRR, abs/21 10.7552 2021. URL
SitEps:/aarxin
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Den- ton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Sali- mans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 36479-36494. Curran Associates, Inc., 2022. URL httpps://proeedding nwwrips.eeieeeee. cc http://ppeeeeeeeeeppeeeeeeeeeeee files/paper/2022 [BBeecCShaaaaa0023aaaaaaaa00aaee
In S. Koyejo, S. Mo-
pdf
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Tiirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. BRLlittps://opeerevee http:/wwwwwwweeeeeeeeeeeeeeeeeee forum 3i:.:mmmmdiMM...... wtt.:eeteooommmoo////
vmsMcY
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6048-608 6688,, June 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under- standing and mitigating copying in diffusion models. arXiv preprint arXiv:2305.20088. 2023b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. CoRR, abs/2010.02502, 2020. URL htttaps://arxi..or/////./
02502
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting stable diffusion using cross attention. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5644-560 5659, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653/v1 1/2023.acl- acI-long..110.. URL https: aaaaacaann hataa/aaaanttoooooooooo org/
acc a ong.310

---

Preprint
Aaron den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL htw.:ttpps/pprrceeeedee.e.errrra
files/
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Causal mediation analysis for interpreting neural NLP: the case of gender bias. CoRR, abs/2004. 12265, 2020. URI. hatttps:/aariv..orgg//o
12265
Ben Wang and Aran Komatsuzaki. GPT-J-6B: Billion Parameter Autoregressive Language Model.
May 2021.
12


Reviews:
[{"review_id": "PU0aIcvisb", "reviewer": "Reviewer_TjyF", "summary": "This paper first introduces the authors' discovery of what are the critical layers for a text to image diffusion model to generate images of certain visual concepts and styles. Based on the analysis, authors propose a method called \"DIFF-QUICKFIX\" that removes/edits certain visual concepts that the model is able to generate.\n\nThe method for localizing these critical layers are adapted from Casual Mediation Analysis, where certain activations of a corrupted model is replaced with that of the original model to identify its impact to the generated image. Specifically both the UNet and text encoder are studied, and experiments show the inspected visual concepts and attributes are scattered on various layers of UNet, as well as the first transformer layer of the CLIP text encoder.\n\nBased on the findings above, authors propose to remove/update concepts by modifying the out put projection unit in the first self attention layer to remap the generated activation to target concept's activation, thus removing/updating the model's generated content.", "questions": "My main question is how the results of this study generalize to other text-to-image generation models, and what are the components that can be reused if we want to apply the method to another model.\n\nOther than optimizing the W_out, an alternative approach would be to directly replace the token embeddings, e.g., from embeddings for \"Van Gogh\" to those for \"painting\". I wonder what the authors' view is on the effectiveness of this approach.\n\nI don't see any results on removing trademarked objects as claimed at the top of page 3. Were they removed during the preparation of the manuscript?\n\nWhile it seems reasonable to use CLIPScore as AIE, it does not seem that |CLIPScore(x_0^{restored}, c) - CLIPScore(x_{0}^{corr}, c)| can be approximated by CLIPScore(x_0^{restored}, c). In particular, both increases and decreases of CLIPScore(x_0^{restored}, c) by the same amount around CLIPScore(x_{0}^{corr}, c) lead to the same absolute difference, but the CLIPScore(x_0^{restored}, c) is obviously different.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method for locating and editing knowledge in diffusion models is a novel approach.\n\nThe editing method introduced in this paper is unique and significantly faster than traditional training-based techniques. As demonstrated by the results presented in the appendix, the method effectively removes or modifies unwanted concepts introduced by the diffusion model.\n\nThe authors clearly convey the paper's main idea, with appropriate background information at most places. The extensive results provided in the appendix are particularly valuable given the lack of well-established metrics for evaluating open-domain text-to-image generation models. These results allow reviewers and readers to effectively assess the method's soundness and effectiveness.", "weaknesses": "The scope of the claims seems too broad. The title and introduction claim to locate the knowledge of text-to-image diffusion models, while in the paper, only one stable diffusion model checkpoint is investigated. Given that most of the findings on this model are through laborious experiments, it is unclear if these findings can be generalized to even other versions of stable diffusion models, not to mention other types of text-to-image models. If the findings are only applicable to the studied checkpoint, the impact of the method may be significantly restricted, as the studied model is not considered state-of-the-art.\n\nLack of comparison to other methods. The visualized results are definitely pleasing to the eye, but it would be better if results of other models could be visualized side-by-side to provide more references.\n\nThe subsection on \"Selecting Threshold for CLIP-Score\" can be confusing to read at the beginning. Perhaps some context of why this is needed would be helpful for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJqeCVV20K", "reviewer": "Reviewer_Qvyq", "summary": "The paper examines the localization of knowledge in language-text models for image generation, specifically UNet and stable diffusion with the CLIP-ViT-L/336px text-encoder. Previous works have not explored these models, and the paper presents some interesting findings such as the wide dispersion of visual concepts across UNet layers coupled with the concentration of text knowledge in a single encoding layer. The latter result is exploited in a new editing algorithm that directly adjusts the weights in the localized component, resulting in high efficiency with similar performance.\n\nThe paper is generally clear, the topic is very timely and significant, and the experiments are reasonable.", "questions": "As above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The use of causal mediation analysis is a good idea, and seems to provide a basis for knowledge tracing, but the text should be clearer about how the approach actually uses or follows the CMA paradigm. It is mentioned in the introductory sections but not later on.\n\nThe knowledge analysis provides significant insight into multi-modal models, showing they seem to store knowledge differently from language-only models. \n\nKnowledge localization analysis is used effectively to enable highly efficient control and editing of the image generation process, through direct model parameter adjustment without any model training. This seems to be a unique advantage of this approach that is reminiscent of manipulating eigenvectors to generate a range of plausible face images (pre-deep learning).\n\nIt seems that the relative importance of each model component is calculated by corrupting it with gaussian noise; generating an image from the corrupted model; then using CLIP-Score to measure the difference between the image and its caption. A low score indicates incompatibility, and implies that the model component is important relative to the caption. This process is linear in the number of model components being tested, and therefore does not scale well to fine-grained components such as individual neurons.\n\nDiff-QuickFix is a clever way to edit generated images, by directly optimizing the weight matrix of a single layer in the text-encoder rather than standard model updating. It is much faster, and appears to be quite effective based on the provided qualitative results.\nThe experimental design seems sound, comparing against two recent editing methods on the same prompt dataset they used previously, via the CLIP-score metric. The results show that the method achieves equivalent editing performance to the baselines, with a huge gain in computational efficiency.", "weaknesses": "The intro is unclear about key points, such as what forms of visual knowledge the paper is focused on, because it is unclear what “visual attribute” means in this paper. In computer vison, an attribute is usually a property of an object such as its color, texture, gender (for a person), presence of accessories (eyeglasses, hats, etc.), age, and so on. It seems that visual attribute here means any sort of visual information, which is confusing.\n\nFig. 2, which is very effective and interesting, shows that there is a large overlap between model components that are causal for the four different attributes. There are a few components (Unet layers) that are unique to different attributes, but most seem to be causal for all attributes, which is problematic. Not only is information distributed widely across the layers for all attributes, but it is largely the same layers that seem to be encoding most information. This is not a weakness of the causal tracing approach per se, but it does call into question the proposed editing method. How effective could it be, when very few model components independently control different attributes?\n\nFig. 3 seems highly unlikely and may call the approach into question, as the illustrated result strongly implies that only one layer and one token in that layer encodes relevant knowledge. This seems highly unlikely. Is there some way to corroborate this result?\n\nMost of the references lack information on the venues in which the works were published. This is unusual and improper for a research paper, although it should be easy to fix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0J03mpznkG", "reviewer": "Reviewer_cjqU", "summary": "The paper studies the source of knowledge about scenes being generated within the architectures of text-to-image diffusion models. It draws the conclusion that such knowledge is distributed within the conditional UNet. It further introduces an image editing method that provides speedup compared to prior arts.", "questions": "* The paper highlights the difference in causal state locations between Stable-Diffusion and GPT. Is such a difference specific to the CLIP encoder or is general to the diffusion model class?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper adapts Causal Mediation Analysis to interpret text-to-image diffusion models and draw several meaningful conclusions in terms of the location of visual knowledge within these models. \n* The paper proposes an editing method based on the observations as an application that achieves empirical advantages compared to prior methods.", "weaknesses": "* The main toolbox of the interpretation method, Causal Mediation Analysis, is borrowed from previous works. There is a limited novelty in terms of the interpretation framework. \n* The experiments presented in the paper all use Stable-Diffusion. The results would be more convincing if other classes of diffusion models could be investigated, which would provide important cues on whether the observations are specific to the Stable-Diffusion architecture or can be transferred to other models that adopt diffusion-based training. \n* The causal states are retrieved solely relying on CLIP-Score, but it's possible that such a score is sensitive to only some prominent visual attributes such as colors. The paper claims to identify the knowledge source of general visual attributes but does not provide an investigation on the efficacy of CLIP-Score to discriminate different kinds of visual attributes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vdqbjcGgtk", "reviewer": "Reviewer_cqRW", "summary": "This paper explores how knowledge corresponding to distinct visual attributes is stored in large-scale text-to-image diffusion models. The authors apply Causal Mediation Analysis to trace the various attributes in U-Net and text-encoder of diffusion models. In addition, the authors introduces an editing method, DIFF-QUICKFIX to edit the image based on their proposed method. DIFF-QUICKFIX can remove or update concepts in images.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overview, the paper is well-written and the motivation is clear. Extensive experiments are performed to demonstrate the meanings of exploring how the attribute information is reflected in the embeddings of text-to-image generative models.\nThough I'm not an expert in causal mediation analysis, I enjoy reading the paper and be happy with the abundant visualization results in the paper.", "weaknesses": "It would be nice if the authors can include more discussions on the limitations of the DIFF-QUICKFIX parts. *E.g.*, what can DIFF-QUICKFIX do in editing tasks. When I check the **remove** results generated by DIFF-QUICKFIX, such as \"Snoppy\" in Figure 5, it seems the \"Snoppy\" is replaced by a dog rather than removed from the figure. From my view, it is more like attribute editing rather than object removal. Is there any potential methods that can leverage DIFF-QUICKFIX to achieve various editing tasks such as remove object, add object rather than simply editing the attributes of existing objects?\n\nIn addition, it would be nice if the authors can add some descriptions in Figure captions. *E.g.*, in Figure 6, I'm confused between \"photo of a apple in a beach\" and \"photo of a apple in a city\". Are there any clues that can help readers to understand why one figure corresponds to \"beach\" and the other corresponds to \"city\"? Maybe add descriptions just like Figure 5 would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)