The Hidden Language of Diffusion Models
Authors: Hila Chefer
Oran Lang
Mor Geva
Volodymyr Polosukhin
Assaf Shocher
michal Irani
Inbar Mosseri
Lior Wolf
Venue: ICLR 2024 poster (2024)
Keywords: Interpretability
Explainability
Text-to-image models
Primary area: visualization or interpretation of learned representations

Abstract: Text-to-image diffusion models have demonstrated an unparalleled ability to generate high-quality, diverse images from a textual prompt. However, the internal representations learned by these models remain an enigma. In this work, we present Conceptor, a novel method to interpret the internal representation of a textual concept by a diffusion model. This interpretation is obtained by decomposing the concept into a small set of human-interpretable textual elements. Applied over the state-of-the-art Stable Diffusion model, Conceptor reveals non-trivial structures in the representations of concepts. For example, we find surprising visual connections between concepts, that transcend their textual semantics. We additionally discover concepts that rely on mixtures of exemplars, biases, renowned artistic styles, or a simultaneous fusion of multiple meanings of the concept.
Through a large battery of experiments, we demonstrate Conceptor's ability to provide meaningful, robust, and faithful decompositions for a wide variety of abstract, concrete, and complex textual concepts, while allowing to naturally connect each decomposition element to its corresponding visual impact on the generated images.

Published as a conference paper at ICLR 2024
THE HIDDEN LANGUAGE OF DIFFUSION MODELS
Hila Chefer 1.2
Oran Lang'
Mor Geva
Volodymyr Polosukhin'
Assaf Shocher'
Michal Irani
Inbar Mosseri'
Lior Wolf
'Google Research
Tel-AAviv University
*Google DeepMind
*Weizmann Institute
https://hm che www.nllaeeeeeergtttttee.be https://hile-chfeer..gii
io/Conceptor/
(1)
Concept decomposition with CONCEFTOR
(b)
Single-image decomposition with CONCEPTOR
artist
abstract
aketching
paint
Pleasso
brushes
CONCEPTOR
Pollook
Van Gogh
ladáer
COSCEPTOR
Monet
portrait
Concept: swjeet peppers
fingers
pepper
staining
studio
aereno
teal
Impressionism
CONCEPTOR
caligraphy
crafreman
glded
Concept: painter
Concept: beeele
hornet
amerald
Figure 1: Concept interpretation with CONCEPTOR. (a) Given set of representative concept images, CONCEPTOR learns to decompose the concept into weighted combination of interpretable elements (font sizes indicate weights). The decomposition exposes interesting behaviors such reliance on prominent painters and renowned artistic styles (e.g., "Monet", "Impressionism" "). (b) Given specific generated image, CONCEPTOR extracts its primary contributing elements, revealing surprising visual connections (e.g., "sweet peppers" are linked to "fingers" due to their common shape).
ABSTRACT
Text-to-image diffusion models have demonstrated an unparalleled ability gen- erate high-quality, diverse images from textual prompt. However, the internal representations learned by these models remain enigma. In this work, present CONCEPTOR, a novel method to interpret the internal representation concept by diffusion model. This interpretation is obtained by decomposing the concept into a small set of numan-interprede state-of-the-art Stable Diffusion model, CONCEPTOR reveals non-trivial structures in the representations of concepts. For example, we find surprising visual connec- tions between concepts, that transcend their textual semantics. We additionally discover concepts that rely on mixtures of exemplars, biases, renowned artistic styles, or simultaneous fusion multiple meanings of the concept. Through large battery of experiments, we demonstrate CONCEPTOR ability to provide meaningful, robust, and faithful decompositions for a wide variety of abstract, concrete, and complex textual concepts, while allowing to naturally connect each decomposition element to corresponding visual impact on the generated images.
textual
textual elements. Applied over the
INTRODUCTION
Generative models have demonstrated unprecedented capabilities to create high-quality, diverse imagery based on textual descriptions (Balaji et al., [2022] Gafni et al. 2022 Ramesh et al. 2021
Rombach al. 2022 Saharia al. 2022). While revolutionary. recent works have demonstrated that these models often suffer from heavy reliance on biases (Chuang et al.) 2003 Luccioni et al. 2023) and occasionally also data memorization (Carlini et al. 2023 Somepalli et al.) (2022). However, all these works draw conclusions by simple external evaluation of the output images, while research on understanding the internal representations learned by the model remains scarce. Thus, our understanding of these impressive models remains limited.
"The first author performed this work intern Google Research.

---

Published as a conference paper at ICLR 2024
In this work, we aim to develop a method to interpret the inner representations of text-to-image image diffusion models. Our approach draws inspiration from the field of concept-based interpretability (Li)
et al. 2021 Kim al. 2017] Ghorbani et al.) 2019). which proposes to interpret the model's 's decision- making for given input by decompasing it into set of elements that impact the prediction (e-g-, decomposing "cat" 'nto "whiskers", "paws" ", efc.). Under this setting, an "interpretation" can be considered to be a mapping function from the internal state of the model to a set of concepts humans can understand (Kim et al., 2017). Notably, existing approaches for concept-based interpretability are not directly applicable to generative models, and, as we demonstrate, fall short of producing meaningful interpretations for such models. Therefore, we propose a novel method produce concept- based explanations for diffusion models, which leverages their unique structure and properties.
Our method, CONCEPTOR, learns pseudo-token, realized as a combination of interpretable textual elements (Fig. aa). To obtain the pseudo-token, CONCEPTOR trains a neural network to map each word embedding the vocabulary of the model to a corresponding coefficient, with the objective linear combination
of denoising the concept images. The pseudo-token is then constructed of the top vocabulary elements weighted by their learned coefficients. This formulation allows us to exploit both the model's 's powerful ability to link between text and image, and the rich semantic information encapsulated in the text encoder. Through a large battery of experiments, we demonstrate that CONCEPTOR produces decompositions that are meaningful, robust, and faithful the model.
We use CONCEPTOR to analyze the state-of-the-art text-to-image diffusion model, Stable Diffu- sion (Rombach et al. 2022) across various concepts, including concrete (e.g., "a nurse"), abstract (e.g., "affection") ") and complex (e.g., "elegance on plate" ") concepts, as well as homograph concepts (e.g.. "a crane") "). CONCEPTOR reveals many interesting observations about the learned representa- tions. (i) As demonstrated in Fig. [b). CONCEPTOR can be used to decompose a generated image to its own subset of driving elements. We find non-trivial combinations of features that control different visual aspects such as shapes, textures, and colors. (ii) We observe that some concepts such as "a president" " or "a rapper" are represented mostly by exemplars, i.e., well-known instances from the concept, such that the generated images are interpolations of those instances. (iii) We additionally find that consistent with previous work (Rassin et al. 2002), the model learns to mix the multiple meanings of homograph concepts. We expand those findings and discover cases where these meanings are leveraged simultaneously. creating images that mix both meanings in a single object. (iv) Finally, we demonstrate our method's effectiveness in the detection of non-trivial biases.
To conclude, our work makes the following contributions: (i) We present CONCEPTOR, a novel method decompose textual concept into set of interpretable elements. Our method utilizes a unique form of decomposition in which a linear combination is learned as a mapping from the textual embedding space to coefficient. (ii) We demonstrate profound learned connections between concepts that transcend textual correlations. (iii) We discover non-trivial structures in the learned decompositions such as interpolations of exemplars, reliance on renowned artistic styles, and mixing of different meanings of the concept. Finally, (iv) We demonstrate the detection of biases that are not easily observable visually. These observations can help discuss ethical questions on factual basis.
2 RELATED WORK
Text-guided image generation Recently, impressive results were achieved for text-guided image
generation with large-scale auto-regressive models (Ramesh et al. [202]] I Yu et al. (022) and diffusion models (Ramesh al. 2022, Nichol et al. 202 Rombach et al. [202]] /aaaaiia et al., 2022). In the context of text-to-inn text-to-image diffusion models, a related line of work aims to introduce personalized concepts to a pre-trained text-to- -o--geeeee model by learning to map a set of images to a "token" in the text space of the model (Gal et al., 2022 Ruiz et al. 2022; Kumari et al. 2022). Importantly, these methods do not produce decomposable or interpretable information and mostly result in a rigid learned vector that resides outside of the model's distribution (Voynov et al. 2023).
concept-based interpretability A similar analysis to ours was conducted on concept representa-
tions in the context of language models (Patel & Pavlick 2022 Li et al.) 202 1 2023b Lovering &
Pavlick 2022). often through projections to the vocabulary space Geva et al. 2022 2023 Ram et al. 2022). For image classifiers based on CNNs, TCAV (Kim et al. 2017) propose the first boncept-based explainability method by training a linear classifier over user defined concepts. ACE (Ghorbani
et al. 2019) leverages multi-resolution segmentation of the class images and clusters the bottleneck
2

---

Published as a conference paper at ICLR 2024
representations of the crops. ICE (Zhang al. 2020) and CRAFT (Fel al. 2022 apply matrix factorization methods on feature map matrix extracted from a set of patches to obtain the decompo- sition. Note that all these methods are not applicable directly to diffusion models. First, most of the methods are hased on CNN architectures (Ghorbani et al. 2019) with non-negative activations (Zhang) et al. 2020 Fel al. 2022), thus cannot be directly generalized negative feature matrices. Most importantly, all methods above perform concept importance estimation by measuring the shift in prediction by employing saliency methods, both are not trivially applicable to a generative model.
Object representation methods related line of work attempts to learn representations for images that hinge on the objects in the image. [Singh et al. (2018) disentangle the background. shape, and appearance of the image for controllable generation. GIRAFFE (Niemeyer & Geiger (020) incorpo- rates 3D scene representation to improve controllable image generation, Slot Attention (Locatello et al., 2020) and DTI Sprites (Monnier et al. 2021) provide improved decomposition with additional features, and GENESIS- V2 (Engelcke et al. (02T learns object representations without iterative refinement. Note that all of the aforementioned methods only offer decomposition for single image and, therefore fall short of providing concept-level explanations. which is our main objective. In the context of diffusion models, the closest work to ours is that of Liu et al. (2023), which proposes to use diffusion models for concept discovery. Importantly, all methods above provide representations that are based on concrete objects and parts that visible in the image, thus are not applicable to our task of interpretability of internal representations (see Appendix G
Diffusion model interpretahility Shared text-image representations such CLIP (Radford et al. 2021) have been analyzed in the past (Chefer et al. 2021a b) Yun et al.) 2023). However, none of these works has been generalized to generative models. As far as we can ascertain, the closest effort to explaining text-to-image models is a simple visualization of the cross-attention maps (Hertz al. 2022; Han al., 2023] Chefer et al.] 2023). Some works (Rassin et al.) 2022, Carlini et al. 2023) have attempted to investigate the images produced by text-to-image diffusion models, and have even found evidence of memorization (Carlini et al. 2023). However, these works rely entirely on the generated images, and are, therefore, time-consuming and require access to the data. For example cxample/Somepalii et al. (002)) were able to show memorization for less than 2% of the generations with SD. Additionally, these works do not attempt to dissect the model's inner representations.
3 METHOD
We apply our method over the ttae-offthh-aa SD employs denoising diffusion probabilistic 2020) over an input latent vector 2т ~ NN0, 1) T...... 1, the DDPM receives a noised latent vector 2, and produces a less noisy vector 2;-1, which serves as the input to the next step.
Preliminaries: Latent Diffusion Models (LDMss)
Stable Diffusion (SD) model (Rombach et al. 2022).
model (DDPM) (Sohl-Dickstein et al. 2015 Ho et al.
and gradually denoises it. Namely, at each timestep,
During the denoising process, the model is typically conditioned on text encoding for an input prompt P. produced by a frozen CLIP text encoder (Radford et al. 202T), which we denote by C. The text encoder converts the textual prompt P to a sequence of tokens, which can be words, sub-words, or punctuation marks. Then, the encoder's 's vocabulary, V E R.N.d is used to map each token in the prompt an embedding vector w Rd where dl is the embedding dimension of the encoder, and N is the number of tokens in the vocabulary. The DDPM model is trained to minimize the loss,
Cracomstruction
B.,.P..... Pew.P. [||e- - E0(2,, I, C(P))) for, 24 / + VI- CtE,
(2)
where Eg is a trained UNet (Ronneberger et al. 2015), and 0 or < GT < Qg = 1. In other words, during training. the input image is encoded to its corresponding latent vector 2. A noise vector E and a timestep t are drawn randomly. The noise vector E is then added to the latent vector = as specified in Eq. 2 and the UNet is trained to predict the added noise E.
Our goal is to interpret the internal representation of an input concept e in a text-to- image diffusion model Ep. Formally, given prompt 'e for the concept c, we learn a decomposition for the concept using the vocabulary V. This decomposition is realized as a pseudo-token w* & V that is constructed as a weighted combination of a subset of tokens from V, i.e.
CONCEPTOR
w
Og
S.C.
W/ E V, 01 On 2 0
(3)
3

---

Published as a conference paper at ICLR 2024
where << N is a hyperparameter that determines the number of tokens to use in the combination.
Learning the set of n vocabulary elements w, and their associated coefficients ag is done separately for each concept c. We begin by collecting training set T of 100 concept images. These images provide the statistics for the concept features we wish to learn. Next, we construct our learned pseudo-token over the vocabulary V. This construction serves vocabulary V is expressive and diverse, containing roughly 50, 000 tokens, ranging from famous personalities emojis and even names of video games F.rtnit is single token in V). By learning our decomposition over this rich set of candidate concepts, we facilitate a meaningful yet restriccte optimization over semantic,
purposes. First, note that the
information. Thus, our choice of candidate concepts achieves the best of both worlds it is both expressive and interpretable. Second, we make a design choice to optimize the coefficients as a function of the word embeddings. This choice is critical to the success of our method since it utilizes the rich textual embedding space of CLIP determining the coefficients. Effectively, this reduces the optimization problem from optimizing 50, 000 unrelated coefficients to learning mapping from smaller semantic space, R. Specifically, our method assigns a coefficient a for each word embedding w using a learned 2-layer MLP, which takes as input the word embedding vector as follows,
Vw E V a = f(w)
(W((((W(())))
(4)
where a is the ReLU non-linearing (Agarap 2018). and W1. W2 are linear mappings. Based on f. we compute w = En= [(w,)) Note that this pseudo-token is not identical to the output token w since w contains all tokens in V. w" is obtained by the top tokens from w.y as described in Eq. 5
To learn meaningful pseudo-token w", we optimize our MLP to reconstruct the images generated from pc, This choice encourages our pseudo-token to imitate the denoising process of the concept images. We draw a random noise ~ NO0 1) and timestep E (1..... for each image, and noise the images according Eq. We then employ the model's reconstruction objective from Eq. [
As mentioned, wIy considers all the tokens in the vocabulary. However, for better interpretability, we wish to represent the input concept with a small set of n << N tokens. Notate by w1,...., Wn E the tokens with the highest learned coefficients. We add regularization loss to encourage the pseudo-token w to be dominated by these top n tokens, i.e.,
Esparsity 1 cosine (w", ww))
This encourages the pseudo-token w", defined by the top tokens in V. to be semantically similar to wy. which defined by the entire vocabulary. Our overall objective function is, therefore,
L Ceveonstruutton + Asporsity Esparsity:
(6)
In our experiments, we set A Apparsity 0.001, n = 50. At inference time, we employ the MLP on the vocabulary V and consider the top n = 50 tokens to compose w", as specified in Eq. Implementation details and a figure describing our method can be found in Appendix A
Given an image / that was generated by SD for a concept e, we wish determine the subset of the tokens from the decomposition w", that drove the generation of this specific image. This is done via an iterative process over the tokens wy E w" as follows; at cach step, we attempt remove single token from the decomposition, w = =Eijs (iytt;, and generate
Single-image decomposition
the corresponding image I, with the prompt Pu and the same seed. Next, we use CL.IP's image encoder to determine if T, is semantically identical J. If the CLIPP score of the two images higher than 95, we remove wy from w" and continue to the next token. This criterion avoids incorporating tokens whose removal only causes minor non- semantcitic modifications to the image (such as a slight shift in pose). This process is repeated for all tokens in w" until no tokens removed.
4 EXPERIMENTS
Concept-based explanation desiderata Following previous literature on concept-based explain- ability (Ghorbani et al. 2019), we begin by defining a set of desired properties that concept- based explanations for diffusion models should uphold. These properties will be the criteria to evaluate our method. (i) Meaningfulness- Each decomposition element should be semantically meaningful and numan-understandable (ii) Faithfulness- The decomposition should be faithful to the concept
4

---

Published as a conference paper at ICLR 2024
camel
girarit
cashmere
snake
tuisted
gecko
snail
ladybug
winding
rettections of earth sphere
aivilizaiion
diefitian
pharmacist
nutritious
tear
scream
u20
painter
electrician
Impregsion of
bonsai
Monet
brush
Japanese serenity pagoda
sakura
Figure 2: Decompositions of single images by CONCEPTOR. The top rows present images found to contain two elements. The last row shows more complex mixtures by removing element at time. The examples demonstrate the meaningfulness of our learned decompositions.
representation by the model. In other words, the decomposition should reconstruct the features manifested in each of the concept images, and produce images that are in the distribution of the concept images. (iii) Robustness- The decomposition should be independent of the selection of the training set and the initialization. Next, we conduct extensive experiments to demonstrate our method's ability to produce meaningful, faithful, and robust decompositions for diverse types of concepts, Throughout this section, we notate by w the token(s) corresponding to the concept c.
We construct diverse and comprehensive dataset of 188 concepts, comprised of the basic classes from CIFAR-10 (Krizhevsky 2009). list of 28 professions from the Bias in Bios dataset (De- Arteaga et al., 2019). 10 basic emotions and 10 basic actions, all 30 prompts from the website Best 30 Stable Diffusion Prompts for Great Images" F which contains complex prompts that require hierarchical reasoning (e.g.. "Medieval village life" "impression of Japanese serenity") and, finally, we consider 100 random concepts from the ConceptNet (Speer & Havasi] (013) knowledge graph to allow for large-scale evaluation the methods. A full list of all concepts is provided Appendix
Data
4.1 QUALITATIVE RESULTS
The ability to link between textual decomposition elements and their visual manifestation is important to facilitate human understanding and establish meaningfulness. We propose two strategies to obtain these connections. First our single-image decomposition scheme, described in Sec. 3 Second, propose to gradually manipulate the element's 's coefficient to observe the visual changes it induces.
Single-image decomposition Figs. and and Appendix D contain examples of decompositions over images generated by SD. The first three rows of Fig. plresent examples of images that decompose into two elements, while the last row contains more complex concepts that decompose into three elements, where we visualize the removal of one element at a time. Note that the results show non-trivial and profound links between concepts. For example, the "snake" Fig. Iis constructed as "Iwisted gecko". ", the "came!" borrows its skin texture and color from the "cashmere" etc. These

---

Published as a conference paper at ICLR 2024
Impul Image
Input Emage
"shork
Concepii:
*ssork"
'Garigg
Concept, dale
douting
Concept: begs
"pple
Concept: apple
*apple"
-Terdy
ooccep:: proto000
merdyy
Tump
Concept: gresicant
rum
Figure 3: Coefficient manipulation. For each of the listed concepts, we manipulate the coefficient of single element from the decomposition and observe its visual impact the generated image.
Table I: Quantitative evaluation of CONCEPTOR and the baselines.
Method
CLIP pairwiset
LPIPS
FID per concept.
FID entire set.)
Token diversity
BLIP-2 token
66.3 + 16.8
0.60 0.13
218.1 1 93.4
46.6
52.1 + 9.7
BLIP-2 sentence
78.7 = 11.2
0.57 = 0.8
158.6 - 74.0
23.7
65.8 + 1.9
PEZ
79.1 + 9.9
0,56 + 0.06
150.6 = 63.4
18.2
75.9 + 1.2
NMF
80.0 = 9.7
0.53 = 0.06
147.0 68.9
21.6
k k-means
82.5. 8.4
0.53 0.05
132.5 1:
21.3
PCA
83.0 4 8.0
0.53 1 0.09
130.8 1: 52.9
19.8
CONCEPTOR
86.2 - 8.3
0.44 0.09
109.5 = 51.8
9.8
69.8 + 3.4
examples demonstrate associations beyond textual correlations, based visual similarities such as shape, texture, and color. Additionally, note that the decompositions represent various different representation strategies. Some involve mixture of shape and appearance (e.g., "snail"), others involve compositional features added gradually (e.g.. "impression of Japanese serenity"), etc. Finally, observe that the "painter" decomposition demonstrates reliance on renowned artistic styles, such that when "Moner" is removed, the painting disappears, even though "Moner" did not appear explicitly in the input prompt. Appendix D. presents a further investigation of this reliance on artistic styles.
Coefficient manipulation Fig. presents examples of coefficient manipulations. First, we present examples of homograph concepts (first, second row of Fig. Observe that some cases, egg. "big apple", both meanings are generated separately in the image (an apple, New York City), while other cases, e.g., "crane", "date" generate a single object. Even in the latter case, the manipulation shows that both meanings impact the generated image, implicitly. For example, when reducing the element "stork" " from "crane", the structure of the crane changes. Evidently, the model employs both meanings simultancously. borrowing the appearance from the machine and the shape from the bird.
Next, we present examples of element inspection. In the first example, we visualize the impact of "nerdy" the concept "professor". As can be observed, it controls the professor's baldness, the glasses, the suit, and the library in the background. Secondly, we inspect exemplar interpolations. Observe that the element "Trump" in an image of "president" controls semantic features borrowed from the identity of Donald Trump while removing this element results in a president that resembles Obama. This example directly demonstrates Obama and Trump are employed sinultaneously in the image generation process. This phenomenon suggests that diffusion models can also memorize by mixing inputs, beyond exact single-sample reconstruction. Further investigation of exemplar based representations can be found in Appendix
interpolation of exemplars. Features from both
4.2 QUANTITATIVE RESULTS
Baselines As far as we can ascertain, our work the first to tackle concept representations in eext-to-image diffusion models. We. therefore, compare method with reasonable baselines and adaptations existing methods. First, we consider a prompt tuning method, Hard Prompts Made Easy (PEZ) (Wen et al. 2023). which aims learn prompt that will reproduce input

---

silver, chesterfield, plate, vegetable, plating, chef, beauty, dessert, saucer, porcelain Published as a conference paper at ICLR 2024
Table 2: Qualitative comparison of the decompositions by our method and the leading baselines.
Concept
PEZ
BLIP-2 sentence
CONCEPTOR
Rapper
marin, prodigy, sturridge, noneeminem
man, hat, shirt
Tupac, Drake, Khalifa, Weekend, Khaled, hood, hat
Medieval
moderated, humpday, giftideas, shistory
people, medieval clothing, village
caravan, peasant, medieval, countryside, farming
village life
Elegance on plate
dinner, moschoizitalk meat
set of training images. Second, we consider two baselines that leverage the state-of- tit-hhh--att art image captioning model BLIP-2 (Li et al. (223a): (i) BLIP-2 sentence extracts a single caption per concept by decoding the mean CLIP embedding of the training images. (ii) BIII-22 token creates one caption per image and constructs a single pseudo-token from the captions, where each token is weighted by its frequency in the captions. Finally, consider CNN-based concept interpretability methods. As mentioned in Sec. 2 these methods are not directly applicable to diffusion models. Therefore, we present comparisons to the closest adaptations of these methods. We decompose the activation of the UNet features from different denoising steps into concept activation vectors (CAVs) using k-means (Ghorbani et al. 2019. PCA (Zhang et al.) 2020 and NMF (Fel et al. 2022). At inference, we project the intermediate activations into the learned space, see Appendix A. for more details.
Metrics For each concept, we test the faithfulness and the diversity of the decompositions. We use test set of 100 seeds to generate images with w' and with each method. Then, we employ three types of metrics: (i) Pairwise Similarity, to measure the faithfulness of the decomposition w.r.t. each of the concept images. We report the mean CLIP (Radford et al.) 2021) image similarity and the LPIPS (Zhang et al. 2018) score. (ii) Distribution similarity, to measure the faithfulness of the decomposition w.r.t. the concept distribution. We report the FID (Heusel et al.] (017)) score with respect to the concept images for each concept separately (FID per concept) and for all concepts as a dataset (FID entire set). (iii) We employ SentenceBERT (Reimers & Gurevych 2019) to measure the element diversity by estimating the dissimilarity of the tokens in the decomposition (Token diversity). This metric further substantiates the meaningfulness by showing that the decomposition is diverse.
The results, averaged across all 188 concepts, are reported in Tab. | As can be seen, our method outperforms all baselines across all faithfulness metrics. Notably, our LPIPS score is at least 10% lower than that of all baselines, indicating that our method is faithful to the concept images. Importantly, both FID metrics obtained by our method are the lowest by big margin (at least 20 per concept and 8 for the entire set). The scales of the FID) scores are very different since the per concept FID was calculated on sets of merely 100 images. However, we report both metries for completeness. Focusing on the more reliable FID score on the entire set, we calculate the "ground-truth" FID between the train and test set and obtain a score of 7.1, which is fairly close to our score. These results establish CONCEPTOR'S 's ability to provide faithful decompositions. Considering the scores by BentenceBERT, CONCEPTOR is only superseded by PEZ. However, PEZ produces a significant amount of uninterpretable tokens (see Tab. 2). The matrix factorization methods do not produce text, therefore we cannot compute the Sentence score. Instead, we enclose in Appendix the top principal components learned for the concepts from Fig. As can be seen, the obtained components do not appear to be coherent or interpretable, ie.. these methods violate the meaningfulness criterion.
Results
Next, we conduct qualitative comparisons between CONCEPTOR and the leading baselines. Tab. 2 compares the textual decompositions, showing that CONCEPTOR learns diverse and meaningful decompositions. Some concepts, such as "a rapper" are dominated by exemplars (e.g. "Drake" "Tupac"), "). while others, such as "Medival village life", are composition of semantically related concepts (e.g.. "caravan". ". "peasant"). In contrast, the baselines either produce decompositions that are not interpretable, i.e., violate the meaningfulless criterion (PEZ), or are oversimplistic (BLIP-2). Please refer to Appendix FF for word cloud visualizations of CONCEPTOR over complex prompts. Fig. prrsents visual comparison to the leading basclines given the same seeds. As can be observed, CONCEPTOR successfully preserves the image features (i.e., upholds faithfuiness), even when the concept entails detailed features. For example, the "painter" images demonstrate reconstruction of the paintings. Conversely, the baseline methods do not accurately embody all features of the concept.
User study To further substantiate our meaningfulness and humans' ability to understand our decompositions, we conduct a user study. In the study, we randomly draw 8 concepts, 2 from each of our data categories: (a) professions, (b) abstract concepts, (c) basic concepts (ConceptNet,
7

---

Published as a conference paper at ICLR 2024
Original Images
Ours
PEZ
PCA
markelplace edlors of Marmkach Figure 4: Feature reconstruction comparison to the leading baselines. For each concept (column) generate the images using the same random noise with our method and the leading baselines, and compare the original concept images generated by Stable Diffusion (Original Images).
poineer
hird
happiness
Table 3: Ablation study of our method, conducted on the professions subset (De De-Aatt et al.
2019
Method
CL.IP pairwiset
IPIPS.
FID per concept..
Token diversity?
CONCEPTOR
87.0 + 5.5
0.45 = 0.07
107.96 = 31.0
69.7 3.4
w/o MLP
78.0 + 6.7
0.55 = 0.06
142.88 45.1
75.9 : 3.0
w/o Eq. 5
80.3 + 11.6
0.52 0.09
146.4 + 63.4
73.2 + 2.1
10
82.9 + 7.8
0.49
129.41 55.3
54.6 + 9.4
100
85.6 + 6.9
0.47 0.47
114.36 39.7
72.8 + 1.8
CLIP top words
80.1 + 9.9
0,513 0.1
130.9 + 57.2
66.3 1 3.9
CIFAR-10), and (d) complex concepts. For each concept, the users were presented with 6 random concept images and 3 possible textual decompositions, as provided by CONCEPTOR, and the two leading baselines that extract text (BLIP-2 sentence and PEZ). The users were asked to select the decomposition that best captures all the features presented the images. Overall, we collected 160 responses for all questions. Of those 160 responses, 70% favored the decomposition by CONCEPTOR above all alternatives, 26.9% favored the captioning by BLIP-2, and 3.1% selected PEZ, further establishing CONCEPTOR 's meaningfulness, even compared to the natural captioning alternative.
Appendix we conduct extensive robustness tests, to demonstrate two important properties: (i) CONCEPTOR is robust to different choices of the training set and initialization, and (ii) CONCEPTOR generalizes the training task to test images. Experiment (i) employs 3 different training sets and initializations and tests the exact match of tokens between the 3 resulting decompositions. The results demonstrate that the decomposition consistent across all sets- 72 80% of the top-10 elements, and 63 70% of the top-25 elements are preserved across all choices. Experiment (ii) shows that w" is able to denoise test concept images from any denoising step, thus w" indeed captures the concept features beyond the selection of training images.
Robustness experiments
In conclusion, we find that the baselines either violate the meaningfulness criterion, i.e., produce uninterpretable decompositions (k-means, PCA, NMF, PEZ, Tab. and Appendix [[] user study) or the faithfulness criterion (BLIP-2, see poor performance in Tab. [ We note that for both criteria and in all experiments, CONCEPTOR significantly outperforms all baselines and obtains robust. faithfull and meaningful decompositions (see Tabs. and
Fig. and Appendix C user study).
We conduct an ablation study to examine the impact of each component on our method. First, we ablate the choice of employing an MLP to learn the coefficients and instead learn them directly. Next, we ablate our sparsity loss and the choice of n 50. Last, we ablate our choice of vocabulary V and instead extract the top 50 tokens by their CLIP similarity to the mean image.
Alation Study
The results are listed in Tab. 3 Replacing the MLP with a vector of weights detrimental to all faithfulness metrics. This demonstrates the importance of our learned MLP, as it leverages the rich semantic information learned by CLIP, rather than optimizing huge set of coefficients without any semantic understanding. Without the sparsity loss (Eq. ()). the top 50 tokens do not necessarily reflect the learned token wyy, and all metrics except for token diversity deteriorate. Additionally, observe that the performance decreases when employing enough to represent all features. For n = 100, the results are similar to the full method, other than the diversity which improves little. This indicates that CONCEPTOR is relatively stable this parameter.
10 since the decomposition is not rich
8

---

Published as a conference paper at ICLR 2024
Input Image
Single Token
Input Image
Single Token
-Ink'
Concept: bridge
+link'
link "cccoruf' Concept: palm free +'cpcoruf' coconut
Figure 5: Examples of elements that impact the generation differently from their standalone meaning.
Finally, when only considering the top words by CLIP similarity the images, the performance decreases substantially, supporting the reliance of our method on wide variety of tokens from the vocabulary, beyond the ones most correlated with the images in terms of textual semantics.
4.3 BIAS DISCOVERY AND MITIGATION
Another important capability of our method bias discovery. Text-to-image models have demon- strated the representation social biases (Luccioni
Table Biases revealed by CONCEPTOR.
Concept
Examples biased terms
al. 2023). CONCEPTOR can be used to discover such biases through the decomposition elements. Tab. lists examples of such biased concepts. Note that our method detects behaviors that are not nec- essarily observable visually such connection between "Jews" " and "journalists". These findings underscore the importance researching concept representations in text-to-image models, biases can impact the generation, even if visually elusive. Utilizing our method, one can also choose to generate debiased versions these concepts by decreasing coefficients associated with biased tokens while preserving other features in the scene. Please refer to Appendix H for such examples.
Secretary
womens, girl, ladies obese, overweight fat
Opera singer
Pastor
Nigerian, gospel refugee, jews mi millennials, blonde
Journalist
Drinking
DISCUSSION AND LIMITATIONS
While method provides faithful and interpretable concept decompositions, there are some lim- itations to consider. First, we find that the visual impact of an element not always completely aligned with its impact as a single token, i.e., the impact of each token depends on the context of the decomposition. Fig. 5 demonstrates such cases. For each concept, we visualize the effect of manipulating the token, and the result of keeping only the token of interest (Single Token). Note that the influence of token on the generated image differs from its influence as a sole token. For example, the token "link" on own produces an image of the video game character, Link. However, in the context of a "bridge", it adds a solid bridge (a link) between the two edges of the image.
Second, our method limited to elements that are single tokens, therefore a complex phrase (e.g., "palm tree") will not be included as single element in our decomposition. However, as mentioned, our construction of the linear combination mitigates the impact of this limitation. Since the context of the decomposition can change the meaning of an element, complex relations can be formed between single tokens by leveraging the context. This is exemplified by our ability to decompose complex concepts, that require hierarchical reasoning (e.g., "elegance plate", "rainbow dewdrops")
CONCLUSIONS
How does generative model perceive the world? Focusing on text-to-image diffusion models. we investigate the model's 's internal knowledge of real- woaldworld concepts. We present CONCEPTOR, method to provide numan--nderstandahhe decomposition for a textual concept. Through extensive experiments, we show that CONCEPTOR provides interpretations that are meaningful, robust, and faithful to the concept representation by the model. Using CONCEPTOR, we obtain various interesting observations the learned concept representations. Via a per-image decomposition scheme, we observe non-trivial connections between concepts in ways that transcend the lexical meaning of the tokens. Furthermore, mixing dual meanings of concepts, or non-tre aniitrivil biases. In all cases, the novel paradigm allows us to shed new light model that, similar other foundation models, still be considered enigma.
method exposes less intuitive behaviors such as the reliance exemplars,

---

Published as a conference paper at ICLR 2024
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu). ArXiv, abs/1803.08375, 2018.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu, ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. ArXiv, abs/221 1..0124,, 2022.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Schwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. ArXiv, abs/2301. 13188, 2023.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782-791, June 2021a.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-mode explainability for interpreting bi-ml and encoder-decoder transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 397-406, October b..
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, 2023.
Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.000700, 2023.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: case study of semantic representation bias in a high-stakes setting. 2019. doi: 10.1145/3287560.
3287572.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. In Neural Information Processing Systems, 2021.
URL JtRL:hhtps://aeg semanticscholoe.... wrrhooaar... org (ttg://oppSSDD:2330000
Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, R R'emi Cadene, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2711-2722 2022. URL http:///.ppp..eeaaatac.cccoooooo
2533708233
Thomas Fel, Victor Boutin, Mazda Moayeri, Rémi Cadène, Louis Béthune, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach unifying automatic concept extraction and concept importance estimation. CORR, abs/2306607304 2023. doi: (0.485500/aXXi.2236607523 07304
URL htt.httppss/ddo..r//100...
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.15 13131, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprinl arXiv:2208.0166
2022.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer ford-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htRttt 2022 wmn.mmmllmmmmmm mamp-mmann3...
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14 147677 2023.
Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. Towards automatic concept-
based explanations. In Neural Information Processing Systems, 2019. URL https::/app semant cscho lart www.org/cooo ttt 184487319
10

---

Published as a conference paper at ICLR 2024
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, arXiv preprint ar arXiv:2303.11305, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Been Kim, Martin Wattenberg. Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference Muchine Learning, 2017. URL https semant
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-conceptt customization of text-to-image diffusion. arXiv preprini urXiv:2212.044880 2022.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 1 11th International Joint Conference on Natural Lunguage Processing (Volume 1: Long Papers), pp. 1813-1827, Online, August 2021. Association for Computational o
Linguistics. doi: 10. 10.18653/v1/2021 http://wwo/222..aloooog...g// acl.-llng..43.... URL ht
2021. acl- long..n..... 143.
Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023a.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wat- tenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference Learning Representations, 2023b. URL https:://ppeeee
net htte//oormmmmddeeee Tc2vT
Nan Liu, Yilun Du Shuang Li, Joshua B. Tenenbaum, and Antonio Torralba. Unsupervised composi- tional concepts discovery with text-to-image generative models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 2085-2095, October 2023.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. ArXiv, abs/2006.150555 2020. URL httph: http://///aaismmaaiiiiaeeieeeeee www.nticscchiieecee.....mcm 220127924
wwgg/oooo org/CorpusID;
Charles Lovering and Ellie Pavlick. Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguistics, 10: 1208 2022. doi: 10.1162/tacl 00514.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias:
Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303./1408, 2023.
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry. Unsupervised layered image decom- position into object prototypes. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.
2021. URL pattps://aphe www.eeettiiicccc...oo.ooooooooee
233476260
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 2021.

---

Published as a conference paper at ICLR 2024
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 11448-11490 2020. URL https: http:/::/.p..smmmmesccchccccccee www.ntieschoaaaattt.eeeeieaceaae org/
CorpusTD:22711 20:221222715557
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022, URL [
net net//Eorum?11c--JUMMMB
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Belinkov, Jonathan Berant, and Amir Globerson. What are you token about? dense retrieval as distributions over the vocabulary. arXiv preprint 11X1:22212.10380, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, 2022. URL https::/arxiv.oorg 23bs/2004.o 06125
Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 seeing double: Flaws in word-to-concept mapping in text2image models, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. https www.i.owwgaaaaaa..aaaaaaoo 9008
10084
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695,, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12.8..22222 12242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, ArXiv, abs/2205.1 11487, 2022.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. 2019 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6483-6492, 2018. URL https
7/api. www.eeanntcccchhhse
Jascha Wohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffu- sion art or digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212 03860, 2022.
12

---

Published as a conference paper at ICLR 2024
Robyn Speer and Catherine Havasi. Conceptnet 5: A large semantic network for relational knowledge. In The People's 's Web Meets NLP, 2013. URL https: wwwpp/emmmmtccccc......oo org
CorpusID:4674513
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. P+: Extended textual condition- ing in text-to-image generation, 2023.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. ArXiv, abs/2302.03668, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do vision-language pretrained models learn composable primitive concepts?, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Ruihan Zhang. Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin P. Rubinstein. In- vertible concept-based explanations for cnn models with non-negative concept activation vectors. In AAAI Conference Artificial Intelligence, 2020. Utttttpss //api semant iesaanttcccholll.
org
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
Ereconstruction
Stable Diffusion (SD)
+
E
A photo of president
SD UNet
UNet
+
ElT
+
SD UNet
IT
SD Vocabulary ("house") W1
MLP
f(w1)
A photo of w/y
WNN
f (w,)..
("flower") WN
fWNN)
X sparsity
Figure 6: Illustration of the CONCEPTOR method. Given the concept of interest (e.g., "a president"), we generate 100 concept images. Next, learned MLP network maps each word embedding w; to coefficient fw::), and the pseudo token V constructed as a linear combination of the vocabulary. We then add random noises eº , !lTl to the images, and use the model to predict the noise based on the text "a photo of a <w >.. " We train the ML. with the objective of reconstructing the images (L (Creconstruction) and add a sparsity loss to encourage sparse coefficients (Eeporsity).
Fig. deseribes the pipeline of our method. Given an input concept (e.g.. "a president"), we obtain set of representative concept images. At each optimization iteration, we add random noises to each of the images individually. Our learned MLP operates over the model's vocabulary to produce coefficient for each of the word embeddings, and the resulting pseudo-token is used to denoise the training images. We employ reconstruction loss to encourage the pseudo-toker to contain the concept elements learned by the model and add a sparsity loss to maintain interpretable and stable decomposition.
All of our experiments were conducted using single A100 GPU with 40GB of memory. We train our MLP as specified in Sec. Sec.Blof the main paper with 100 images generated from the concept using seed 1024 for maximum of 500 training steps with batch size of (which is the largest batch size that could fit on our GPU). Additionally, we use a learning rate of le 3 (grid searched on 5 concepts between le 2, le - 3, le 4). We conduct validation every 50 optimization steps on 20 images with validation seed and select the iteration with the best CLIP pairwise similarity between the reconstruction and the concept images. We use the latest Stable Diffusion v2. text-to-image model employing the pre-trained text encoder from the OpenCLIP VIT-H mode with fixed guidance scale of 7.5.
Additionally, to filter out meaningless tokens such punctuation marks consider the vocabulary to be the top 5, 000 tokens by their CLIP similarity to the mean training image. We note that this find
filtering method is fairly coarse, and meaningless tokens remain in the vocabulary, however, that this choice improves the convergence time of our MI. such that 500 iterations are enough to obtain meaningful decompositions. This choice is ablated in the main paper (see the CLIP top words ablation). Our single-image decomposition scheme employs a CLIP ViT-B/32 model (Radford et al. 2021). Please find our code attached as ZIP file reproduce our results.
A.1 CONCEPT-BASER BASELINES IMPLEMENTATION DETAILS
To implement the concept- based baseline approaches, we follow the formulation in Fel et al. (2023). During the generation of the training concept images, we collect the intermediate representations from
ffusion
14

---

Published as a conference paper at ICLR 2024
the UNet different denoising timesteps. Then, decompose those representations using PCA, following Zhang et al. (2020). NMP, following Fel et al. (2022) or k-means, following Ghorbani et al. decomposition size. Finally, at inference,
(2019). We use total of 50 components, to match we generate images from the concept using the test seed, 2, while projecting and reconstructing the aforementioned intermediate representations both before passing them to the next block and the upsampling (decoding) part of the UNet (via the skip-connections), At training and inference when using NMF drop the negative activations. At inference, when using k-means, we employ the closest cluster as reconstruction of an intermediate representation.
We compute PCA using iterations of randomized SVD solver with 10 oversamples.
We compute k-means using 10 k-means++ initialization and 100 iterations of the Lloyd algorithm. We compute NMF using NNDSVD initialization and run the Coordinate Descent solver for 200 iterations.
15

---

Published as a conference paper at ICLR 2024
B DATASET
In the following, we enclose the full list of concepts in each of our data subsets.
Bias in Bios concepts "professor", "physician", "attomney". "photographer", "journalist", "nurse", "psychologis"", "teacher", "dentist", " "surgeon". "architect"". "painter", "model", "poet", "fimmaker", "software engineer", "accountant", "composer", "dietitian", "comedian", "chiro- practor", "pastor", "paralegal", "yoga teacher", "dj", "interior designer", "personal trainer", "rapper"
CIFAR-10 concepts "ship", "truck"
"airplane", "automobile", "bird", "caf", "deer", "dog", "frog", "horse",
Emotions "affection", ", "anger", "disgust", "fear". "happiness", "honor", "joy", "justice", "sad- ness", "beauty"
Actions "clapping", "climbing", "drinking", "hugging", "jummiing" "pouring". "running", "sitting", "throwing", "walking"
Complex concepts "Rainy New York Nights", "The Fashion of Abandoned Places", "Rainbow Dewdrops", "Aerial Autumn River", "Skateboarder's Urban Flight", "Dive into Coral Reefs". "Vintage European Transit", "Star Trails over Mountains", "Marketplace Colors of Marrakesh", "Elegance on Plate", "The Renaissance Astronaut", "The Surreal Floating Island". "Impression of Japanese Serenity". "Jazz in Abstract Colors", "The Confluence of Pop Art", "The Robotic Baroque Battle". "Cubist Bustling Market". "The Romantic Stormy Voyage". "The Botanist in Art Nouveuu", "The Gothic Moonlit Castle", "Neon-Souked Cyberpunk City", "Dragon's Stormy Perch", "Reflections of Earth", "After The Fall", "Retro Gaming Nostalgia". ", "Medieval Village Life", "Samurai and the Mystical", "Minimalistic Geometry", "Alien Flora and Fauna", "The Invento''s Steampunk Workshop"
"curling iron", "baseball stadium", "flowers", "submarine", "policeman", "projectile", "tissue holder", "jogging", "storey", "sickness", "parlor", "ships", "conductor", "booze" " "key", "metal", "prostitute" " "wings", "tools", "road". "main", " "leader". "radio", " "sur- prise". "chips", "castle". "bathroom" ", "compete against". "leather", "science", "rich". "sponge". "bell", "eloquent". "nightclub" " "water", "patient", "eat vegetables", "respect", "lemur". "bum". "mammoth", "birthday", "chain", "cats", "frogs". "arkansas", "basketball", "listening" "dream". "ticket office", "failure", "text", "now". "oven". " "leg", "mundane" " "copulate", "tree", "wood", "mail", "wooden rod". "clippers", "competing against", "dull", "book", "watch television". "win- ning baseball game". " "iphone". "dance club", "security", "politician", "subway station" "fall", "junk", "sleighing ride". "call". "mosquitoes", "national highway", "contraceptive device". "state- ment". "kill", "seeing old things", " "lift", "adults". "pillowcase" "wedding ring", "eyes". "coun- try", "stepladder", "mandolin" "reception area", "'hief"" "plastic", "projector", "hub" "card catalog", "negligible", "rook". "Ilano estacado"
ConceptNet concepts
16

---

Published as a conference paper at ICLR 2024
C ROBUSTNESS EXPERIMENTS
In the following sections, we conduct experiments to demonstrate our method's ability to provide robust concept interpretations. First, we show that the obtained decomposition is stable, i.e., that the same elements are learned across different choices of training sets and initialization. Second, we test our decomposition's 's ability generalize w.r.t. the denoising training task, i.e., test the denoising capabilities of w on test images. This shows that the elements in w represent the entire concept, beyond the training images.
C.I ROBUSTNESS TO TRAINING DATA
Note that since N >> d, there are many linear combinations that yield w". However, due to the specific MLP-based structure and the sparsity constraints, the decomposition is stable across multiple draws of training sets. In this section, we aim to verify that empirically. For each concept, we generate 2 alternative training sets with different random seeds, in addition to our original training set, to test the consistency of our results. For each alternative training set, we decompose the concept using our method as described in Sec. 3 with a different initialization for the MLI This process results in decompositions of n 50 tokens for each concept.
We then analyze the intersection of the top 10, 25, and 50 tokens between the original decompo- sition and each of the alternative decompositions. The concept intersection score for k is defined to be the average of the intersections with the two alternative sets. In other words, we calculate two intersection sizes for k: between the top & tokens of the original decomposition and the first alternative decomposition, and between the top l tokens of the original decomposition and the second alternative. The overall concept intersection score for & is the average of the two. Standard Deviation is computed across the concepts. Note that this experiment measures an exact match of the tokens, therefore the actual intersection may be even higher, e.g., if synonym is used in of the alternatives.
Table 5: Decomposition consistency experiment. For each number of tokens (k = 10, 25, 50) we test the intersection between our learned top tokens and those learned by employing two different training sets of concept images, with different random initializations. The results demonstrate that the top tokens are consistent and robust across different training sets and seeds.
No. of Tokens
Intersection
Abstract Conerete
Top 10
8.03 (80.3%) 2.07
Top 25
17.68 (70.7%) + 4.47
Top 50
28.96 (57.9%) = 8.05
Top 10
7.20 (72.0%) 1.86
Top 25
15.95 (63.8%) + 3.97
Top 50
25.65 (51.3%) + 5.41
The average intersection scores across all concrete concepts and all abstract concepts are presented in Tab. As can be seen, for the concrete concepts, an average of 8.03(80.3%) of the top 10 tokens are present in all the decompositions, even when considering an entirely different training set, indicating that the top tokens obtained by our method are stable. Additionally, when considering the top 25 tokens, an average of 17. 17.68(70) (70.7%% of the tokens are present in all decompositions, which is large majority. We note that the bottom tokens are typically less influential on the decomposition, as they are assigned relatively low coefficients by the MLP. Accordingly, when considering all 50 tokens, an average of 28. 00..6667.999 (57.9%) of the tokens appears in all decompositions. The results for the abstract concepts slightly lower, yet demonstrate similar behavior. Overall, the results demonstrate that our method is relatively robust to different training sets and random seeds, such that even in the face of such changes, the top-ranked tokens remain in the decomposition.
17

---

Published as a conference paper at ICLR 2024
Concept tiken
0ut
Optinized
Input I
0.
-02
-04
Denoising with w
-1.0
AT
Denoising with w²
-1.
202 600 800
Stop
(a)
(b)
Figure 7: Generalization tests comparing the concept prompt, w², and our pseudo-token, w". (a) Quantitative test on all concepts. For each timestep, we add random noises to the images and compare the reconstruction with w", w", and w", a contimuous token optimized with the same objective as w* (Optimized Token). We report the MSE after subtracting the score of random token. (b) Qualitative comparison. An image I is generated from the concept "a murse", and different noises are added E (Ist row). We then compare the denoising with w" (2nd row) and with w" (3rd row).
C.2 DENOISING GENERALIZATION
We wish to ascertain that the learned decomposition w² can generalize its training task to test images. Recall that during training, w" accumulated the elements necessary to denoise the concept images for each denoising step. Next, we wish to demonstrate that these features can perform this same training task on test images. We compare the denoising quality with w" to that of the concept prompt token(s), wº, This comparison serves two purposes: (1) establish CONCEPTOR* ability to learn the actual concept features, beyond a set of features that simply reconstruct the training set, (2) motivate the fact that, as can be seen in our results, w" A w To intuitively motivate the latter, note that even though w was used to generate the concept images, it is not necessarily the best prompt to denoise them, since: (1) wf generates each image using a specific initial random noise, but not guaranteed to be better in demoising them after applying other random noises. (2) Unlike w", w" is constructed as a linear combination of tokens. Thus, our optimization is performed over a larger, continuous space of embeddings, and therefore is more expressive than simple selection of tokens.
We begin with a quantitative comparison. We sample a test set of 100 images for each concept in the dataset. Then, for each denoising step t f1..... .T and each test image, we draw random noise and apply it as in Eq. 2 Finally, we test the reconstruction loss specified in Eq. 1 with the pseudo-token w² compared the concept prompt wº, To provide lower bound the obtainable error, we additionally compare to w", vector optimized with the same reconstruction objective on the entire continuous embedding space Ed without restrictions, similar (Gal al. 2022). Note that, unlike w², w" does not offer interpretable information, since it is non non-decomposablee and often diitriiiiitiiiiiionn (see Sec. D. However, demonstrates our claim that optimization over larger domain yields better reconstruction. Additionally, we observe that there is a large variance in the MSE score across timesteps. Latents in early steps are very noisy, and therefore obtain a very high loss (~ 0.8), while the last steps contain virtually no noise, and the MSE is very low (~ 1)- Therefore, we compute a baseline score to normalize the scale. We subtract from each score the denoising score for the same images using random token which serves as an upper bound for the MSE. Fig. 7 a presents the results averaged across all concepts, showing that the concept wo obtains score worse than both w² and the optimized token wº, which obtains the best results. These differences are statistically significant, as shown by the error hars marked on every timestep. Evidently, by optimizing token over a larger domain, we can outperform the original concept token in the denoising task. Importantly, this motivates the different features learned by w" while also demonstrating the remarkable denoising capability of wt on any concept image, which indicates that w" indeed captures the concept features.
Fig. ())) provides a qualitative comparison between wh and w². An input image I generated by "a photo of nurse" is noised and then denoised back from different denoising steps, using the concept token w and our pseudo-token w". As can be seen, there are cases where, given different random
Our code does not employ fixed random seed, thus each run implics a different random initialization for the MI.P and a different set of random noises and timesteps for the training images.
18

---

Published as a conference paper at ICLR 2024
seed, w" does preserve the features in the original image I (e.g., adds hats, face masks, and black and white effects), while w does. Intuitively, this can be attributed to the rich representation learned by w", which can include both semantic and style features. Both experiments motivate the diversity of the learned decomposition. Since w is not necessarily optimal for Eq. [ w" learns additional features to improve the denoising quality. Thus, w" balances two objectives- interpretability and faithfulness to the model's internal representations.
19

---

Published as a conference paper at ICLR 2024
D SINGLE-IMAGE DECOMPOSITION
In this section, we provide additional examples of single-image decompositions obtained by our method, as described in Sec. of of the main paper. Fig. presents the obtained decompositions for images generated by Stable Diffusion for given concept. As can be seen, the phenomena demonstrated the main paper are reproduced "sweel peppers" borrow the appearance from the "pepper" and the shape from the "fingers", the "frog" borrows the overall appearance of a "toad" with the color of "Kermit", etc. Other concepts constructed composition of related elements, for example, "happiness" is constructed as "children" who are "laughing" ", and a "lamp" a "chandelier" with a "bulb" in it.
the presented decompositions. For example,
nightolub
tumtable
0ee
wasp
honey
corn
maize
comb
sweet peppers
tingers
pepper
oak
stag
sequcia
wardrobe
pantry
robe
snake
QB0KO
dragon's 8 stormy perch
Pagasus
Godzila la
happiness
children
laugh
chair
atool
cello
camel
ostrich
cashmere
Lamb
bulb
chandeler
frog
foad
Kermit
palm free
coconut
mahogany
Figure 8: Decompositions of single images by CONCEPTOR. Each the examples depicts an image generated by Stable Diffusion for the concept, and its corresponding decomposition.
20

---

Published as a conference paper at ICLR 2024
D.1 RELIANCE ON RENOWNED ARTISTIC STYLES
As demonstrated in Figs. and 2 we find that the representation of the concept "painter" relies on the renowned artistic styles famous artists. In this section, further verify this observation through the lens of our single-image decomposition scheme. We generate 100 test images for the concept "painter" and apply our single-image decomposition scheme all images. Out of the tested images, we found that 67 images contain at least one name of famous artist in their decomposition. This result empirically demonstrates the reliance on existing artistic styles, even when the prompt does not specify the artist's name explicitly. Fig. 2 demonstrates the impact of the experiment described above on the generated images. We observe that removing the names of renowned artists modifies the painting generated in the image entirely (top row) or, in some cases, removes it from the image altogether (bottom row).
Original Images
w/o Renowned Painters
Figure 9: Generated images for the concept "painter" before and after removing the names of famous painters from the single-image decomposition. As can observed, this removal results in significant modification to the painting in the image.
21

---

Published as a conference paper at ICLR 2024
E REPRESENTATION BY EXEMPLARS
Tab. Cand Fig. present examples of concepts that rely on famous instances for their representations. For example, "movie star" is represented by names of famous actors such Marilyn Monroe or Lucille Ball. Similarly, the concept "president" is dominated by American presidents such as Obama and Biden, and the concept "basketball player" relies on famous players such as Kobe Bryant and LeBron James. To demonstrate the reliance on the exemplars in the decomposition, Fig. TO shows the
reconstruction by our method with and without the famous instance names from Tab. 6 can be observed, the reconstruction quality heavily relies the identities of the instances, and when we remove those instances the reconstruction is harmed significantly.
Table 6: Exemplar-basce decomposition elements obtained by CONCEPTOR.
Concept
CONCEPTOR
Composer Movie Star
Schubert, Beethoven, Chopin, Mozart, Brahms, Wagner Aubrey, Bourne, Lucille, Gloria, Marilyn, Monroe, Oswald Obama, Trump, Biden, Nixon, Lincoln, Clinton, Washington Tupac, Drake, Khalifa, Weekend, Khaled, Eminem, Wayne Kobe, LeBron, shaq, Bryant, Jordan, Donovan, Kyrie
President
Rapper
Basketball player
Original
Images
Ours
w/o Exemplars
composer
movie star
Original Images
Ours
ARR
w/o Exemplars
president
rapper
Figure 10: Examples of concepts that rely famous instances. When removing the exemplars, the reconstruction quality is significantly harmed.
22

---

Published as a conference paper at ICLR 2024
F DECOMPOSITION OF ABSTRACT AND COMPLEX CONCEPTS
In Fig. 11 we present decomposition results for abstract and complex prompts from our dataset (see list in Appendix in the form of word clouds. As can be observed, our method success- fully produces meaningful decompositions for various abstract concepts including emotions (e.g., "happiness", ", "sadness" as well as complex abstract concepts (e.g. "elegance on a plate"). The decompositions demonstrate intriguing connections between abstract elements (e.g.. "happiness" is linked "childhood", "laughter", "dream" "))
heroes
Smi l e
Ourant
Joy
mother
wi.dos ee on
wini estin
depression ion
wort
alone
soulg
Seenage
perfect
resanco
beautiful
end
relationshi
love
emot
Ineceence
father
Laughter
fami
lychildhood
entent
scchi
crying
loving
kibsin
snuzzle
companion
Bing
le
mate
ppppy
Sapolyyaughing
happy
sour
Laugh
fr
friendshap
saiting
S
sympathy
sranger
Bor
ner
sweetheatt
choos
puppies,
ffieedishi
hepplly
ch
d
r
angels
eam
hope
life
gr
unhappy
offected
miss afraid
anditen anvilety
budd i
pa
gorre gotasss
ther
gr
over
grandpareed
mer.
f
families
people
depressed SaeetterSCared
toge ti
the
adorable
Concept:
Concept: sadness
Concept: affection
happiness
yugure or
awn
choro
plated
azalea
pagoda
a
civilization
plating
knife
per
tabletor
trout
e
cake
aes anberr dos
enlighten
animation
heavenly
Cion
nightenmentt
harmony
Liction
Tuery
deforestation
baby
babylon
S
stump
Inception
Lun
steak
che
Cour ses
Curtery
Deauty
fres
ceent
a
avocado
porcelsing
taurant
W
D
maanneet contr
allure wonde
ar oma t therehr
rentra
b blossome avatar
ar twork ks
oms
apy
sakura
wonders
Land
f floating
fortnite
ttrolll
Floating
wo
tonehenge
issplatoon
sales
ders
S
mistive
a
ess
sel
ponsai
bord
tree
a aamusrousro Snvvesmen:
derder
moa
ands
saucer c cutümbe pastry
tranqui
mas tery
1 inannnt ion
srreeoouse
rerlecting
seas
islanders
Concept:
Concept:
Concept:
elegance on a plate
impression of Japanese serenity
the surreal floating island
jung
sno-
celing
Le
vent
abysa
tuna
coloringa
baal ands
badlands
tourism
force
estro
shapes rakesh
coloring
g
c
purs
llats
corsl
reef
sub
f
underwater
Storm
al
m
wonder land
O
puzzles
crafts
Cassss
spiro
scenery
divers
sege
barbles
fabrac
market
thread
hydraulic
W
wreck
enrona
mon
tana
eco
O
Swi PELLwwar
I
iee
underwor andolu
m
Time
ky
di trays
colourtt to
desert it Reeals
o
marrakech
Reakingg
it's
Leve
Lig
iightning
sparks
belts
canyon
Fricete
Subme
ear serranessuee sunken
ents
colourin ing
rumb
chemtrails
meridian
necklaces
aipür
Concept:
Concept:
Concept:
dive into coral reefs
star trails over mountains
marketplace colors of Marrakech
Figure 11: Decompositions obtained by CONCEPTOR for abstract and complex concepts from dataset.
23

---

Published as a conference paper at ICLR 2024
G OBJECT-CENTRIC REPRESENTATION METHODS
The closest relation we observed between object-centric representation methods and CONCEPTOR through the task of unsupervised object/ concept discovery. Importantly, the task of interpreting diffusion models largely differs from that of object discovery, since an interpretation should often contain elements beyond objects that are physically present in the image. For example, Fig. 2 shows connections such as "snake" to "gecko" + "twisted"" a "gecko" " is not present in the image, and "twisted" is not an object an adjective. Similarly, "cashmere" not present in an image of "camel", etc. In contrast to these semantic, profound connections, interpretation using object discovery only provides representations that are based on concrete objects and parts that are visible in the images and falls short of discovering the additional connections required to interpret the model (e.g.. reliance on exemplars, non-trivial biases, etc.).
To empirically demonstrate this point, Fig. presents exemplary results using (Liu al. 2023), which the state-of-the -he-ttr-rtt concept discovery method based on SD. Fig. [2 performs concept discovery of three exemplary concepts from our dataset, "snail" and "snake" aand complex concept "impression of Japanese serenity". As can be seen (first column for "snail" and "snake" ", fourth for " "impression of Japanese serenity") the method learns to embed the concept in single token (as expected from its task definition), even if the concept is complex, and does not employ any other learned tokens, therefore they are semantically meaningless for interpretability purposes.
Concept
Concept
Concept
Concept
shail
snake
impression of Japanese serenity
Figure Examples of concept discovery with (Liu al., (023) for "snail", "snake" " and "impression of Japanese serenity". As can be observed, the entire prompt is encapsulated in single concept, while other discovered concepts are random. This aligns with the objective of the method, to divide all concepts in the image between the different tokens.
24

---

Published as a conference paper at ICLR 2024
H CONCEPT DEBIASING
As mentioned in the main paper, CONCEPTOR is capable of detecting biases that are otherwise difficult to capture by simply observing the generated images. After detecting bias, could employ coefficient manipulation with our method to mitigate the bias. Fig. 13 presents examples of such concept debiasing for the concepts in Tab. 7 For each concept (row), decrease the coefficient of the biased tokens in the decomposition until an unbiased representation is achieved. Fig. [3] demonstrates a comparison of the generated images by SD without intervention (Original Images), and the images reconstructed by our method after reducing the bias coefficients, on the same set of 8 random seeds. As can observed, our method able to mitigate the different biases while maintaining the other features represented by the concept.
Table 7: Biased terms detected and removed by CONCEPTOR using coefficient manipulation.
Concept
CONCEPTOR
Professor
men
Nurse Secretary
woman, women, wife, mother wifc, hostess, wonens, girl, ladies obese, overweight, fat
Opera singer
Original Images
Debiased
Professor
Nurse
Secretary
Opera singer
Figure 13: Examples of concept debiasing using CONCEPTOR with the same set of 8 random seeds. The first three examples demonstrate gender debiasing, while the last one demonstrates a decoupling between "opera singer" and obesity.
25

---

Published as a conference paper at ICLR 2024
I PRINCIPAL COMPONENT VISUALIZATION FOR BASELINES
In this section, we enclose the concept activation vectors (CAVs) learned by the leading concept-basee explainability method adapted from CNNs (PCA). Fig. 14 present the reconstruction and components learned by PCA. To visualize a component, we remove it and generate the corresponding images without it examine its impact on the generations. As can be seen, most of the learned components are not easily interpretable and do not demonstrate a coherent semantic change across the different test images. Thus, these methods violate the meaningfulness criterion. We note that meaningfulness without it, humans cannot understand the decomposition,
is a critical criterion for interpretation, and it does not serve its basic goal of giving insights into the model.
Original Images
PCA
Component
Component 2
Component 3
Component 4
Component 5
Painter
Bird
Happiness
Marketplace colors of Marrakesh
Figure 14: PCA top extracted principal components for 4 concepts. The first depicts SD's original images, the second row shows the reconstruction by PCA, and the last 5 rows demonstrate the impact of removing each the top 5 principal components learned by PCA.
26

---

Published as a conference paper at ICLR 2024
J OUT OF DOMAIN (OOD) EXPERIMENTS
The main objective of our method is the interpretation of the inner representations of concepts by the model. With that, in this section, explore CONCEPTOR'S ability provide insight into how the model represents out-of-domain concepts.
We present examples of three out-of-domain concepts extracted from the DreamBooth dataset (Ruiz et al. 2022). These examples represent increasing distance from the model's 's distribution. The first concept represents an example of concept that is close the model's distribution, i.e., a specific dog. The second one deviates further from the distribution and portrays a sloth plush. Finally, we present an example of concept that completely out-of-domain, monster doll with unique appearance that the model is not familiar with.
We use CONCEPTOR to interpret the representation each of these concepts and apply a baseline where we mimic CONCEPTOR optimization without any restrictions w" (similar to (Gal et al. 2022 The results for all three concepts are presented in Figs. 15 to 17 respectively.
As can be expected, the reconstruction quality of both the baseline and CONCEPTOR degrades we move further away from the model's distribution. However, interestingly, even considerable OOD cases (i.e., the sloth plush, the monster toy) the decompositions by CONCEPTOR remain meaningful, as is reflected by the decomposition elements presented the bottom two rows of each figure. For example, the sloth plush in Fig. [6]is linked to the concepts "plush", "fluffy", and "knitted" and the monster toy in Fig- 17 is linked to "woolly" due to its wool-like fur, to the character "Domo" due to the shape of its body, and "crab" and "shrimp" due its unique limbs. These results further substantiate our method's powerful ability to learn meaningful decompositions, even in the face of OOD samples.
27

---

Published as a conference paper at ICLR 2024
Training images
Unrestricted opt.
CONCEPTOR (ours)
Top decomposition elements
Corgi
Dachshund
Beagle
Chihuahua
Chihu
Simba
Figure 15: Out domain samples. The first two rows depict the training images from Ruiz et al. (2022). The next depicts samples drawn from the concept learned by unrestricted optimization. The third row depicts samples created from the concept we learned, w*. The last two rows present the primary decomposition elements learned by our method for the concept.
28

---

Published as a conference paper at ICLR 2024
Training images
Unrestricted opt.
CONCEPTOR (ours)
Top decomposition elements
Sloth
Plush
Fluffy
Teddy
Koala
Knitted
Figure 16: Out of domain samples. The first two rows depict the training images from Ruiz et al. (2022). The next depicts samples drawn from the concept learned by unrestricted optimization. The third row depicts samples created from the concept we learned, w*. The last two rows present the primary decomposition elements learned by our method for the concept.
29

---

Published as a conference paper at ICLR 2024
Training images
Unrestricted opt.
CONCEPTOR (ours)
Top decomposition elements
Elmo
Woolly
Domo
Crab
Muppet
Shrimp
depict the training images from Ruiz al. (2022). The next row depicts samples drawn from the concept learned by an unrestricted optimization. The third row depicts samples created from the concept learned, w*. The last two rows present the primary decomposition elements learned by our method for the concept.
Figure 17: Out of domain samples. The first
30

---

Published as a conference paper at ICLR 2024
K ROBUSTNESS TO ALTERNATIVE LOSS FUNCTIONS
In our experiment, we chose to employ the model's reconstruction loss (Eq. 1 to encourage our decomposition encapsulate the features learned by the model for the concept. In this section, explore the possibility of replacing the MSE reconstruction with a semantic similarity engine, i.e., CLIP (Radford et al. 202T). Since the UNet, Es. predicts the added noise (for which CLIP is not directly applicable), we apply our loss over the prediction of the clean image, Tp. iee.
CCLP = 1 cosine (CL) CL)P((
where zo is the clean training image, and do the predicted image by the diffusion model, i.e.:
3t
(8)
where n; is the noised version of the input image zn according to Eq. 2 and is the noise predicted by the UNet. In other words, we calculate the predicted clean image based on the noise prediction and apply the CLIP loss against the original image. This process will result decomposition that gives us the elements best correlated with the features of the concept according to CLIP, We note that since CLIP is a semantic engine, some degradation in the reconstruction performance is to expected. In Tab. 88 and Fig. [8 we present a qualitative comparison between the decomposition and reconstruction obtained by our method with the CLIP loss, compared the original MSE loss used. We employ the same concepts and seeds from Fig. of the main paper. As can be observed, while the reconstruction quality indeed decreases, the same phenomena observed with the MSE loss are reproduced with the CLIP loss. For example, the concept "painter" still heavily relies the same famous artists discovered by our full method (e.g., "Monet", "Picasso") "). These results further substantiate the conclusions presented in the main paper, since it is shown that the method is insensitive the choice of the reconstruction loss.
Table 8: A comparison between the decompositions obtained with CONCEPTOR with MSE loss vs. CLIP image loss.
Concept
CONCEPTOR w/ CL.IP Loss
CONCEPTOR w/ MSE Loss
Painter
Monet, painter, Picasso, watercolor, Impressicnist, photographing, painting...
Monet, Picasso, paint, Impressionism, artist, studio, brushes, sketching...
Bird
kingfisher, bunting, bluebird, starling, magpie, finch, parrot, jays...
teal, beak, brown, stripped, dove, quail, sparrow, hummingbird, finch...
Happiness
smile, warmth, happiness, children, delight, emotion, cheerful, love, friendship...
children, smile, laughter, dream, families, emetion, smile, mother, friendship....
Marketplace flavor, Bangkok, Mumbai, colors of souvenir, fragrances, flavors, market, fabric, desert, oasis, Marrakech
Marrakech, tents, coloring,
tourist, bazaar, spices...
spices, dust, ornaments...
31

---

Published as a conference paper at ICLR 2024
Original Images
Ours w/ MSE
Ours w/ CL.IP
painter
bird
Original Images
Ours w/ MSE
Ours w/ CLIP
happiness
markotplace colors of Marrakech
Figure 18: Feature reconstruction comparison between our method with an MSE loss and with CLIP loss. The original images appear on the first row.
32

---

Published as a conference paper at ICLR 2024
L ROBUSTNESS TO THE MODEL SELECTION
In this section, we conduct an experiment to demonstrate that CONCEPTOR is robust to the choice of the encoder and the model. We apply our method over SD 1.4, which employs different text encoder than that of SD 2.1. Importantly, SD 1.4 uses CLIP ViT-L/14 which was trained on proprietary non-public data, while SD 2.1 uses OpenCLIP- ViT/H, which is an open-sourced version of CLIP, trained on public data. These two text encoders are entirely different not only in training data but also in the size of their word embedding vectors (768 for CLIP ViT-L/14 vs. 1024 VIT/H). We compute the decompositions by CONCEPTOR for the same concepts as those in Fig. 4 in the same setting described in the paper. The qualitative results, both the textual decompositions and the feature reconstruction results, are presented in Tab, 9 and Fig. 19 respectively. As can be seen, the decompositions remain human-underse uumaa--underttaddbble and faithful to the model, as expected. Additionally, to establish that reliance exemplars exists across versions of SD (independently of the training data and text encoder), we enclose in Tab. the decomposition for the concept "president", showing that it is still dominated by former American presidents.
OpenCLIP.
Table 9: Textual decomposition elements obtained by CONCEPTOR for SD 4.1.
Concept
CONCEPTOR
Painter
painting, watcrcolor, sketch, studio, pastels, artist, canvas, ladders, oilpainting, brush...
Bird
avian, birdie, jay, phoebe, pilgrim, blackbird, hummingbird, whistler, peregrine, cardinals...
Happiness
triendships, childrens, rejoice, Love, photography, unforgettable, everyone, dreamers, smiles...
Marketplace colors of Marrakech
marketplace, marrakech, paprika, rugs, terrace, dyes, vendors, barrels, bowls, colorful...
President
potus, barackobama, obama, nixon, washington, resldonaldrrum, jefferson, washington, roosevelt...
Original Images (SD 1.4)
Ours
painter
bird
Original Images (SD 1.4)
Ours
happiness
marketplace colors of Marrakeoh
Figure 19: Feature reconstruction results of our method on SD 1.4.
33


Reviews:
[{"review_id": "WckAh1wxdL", "reviewer": "Reviewer_HDAw", "summary": "The paper delves into understanding the internal representations of text-to-image diffusion models, which have shown significant prowess in generating high-quality images from textual concepts. The primary challenge addressed is deciphering how these models map textual prompts to rich visual representations. The authors introduce a method, \"CONCEPTOR\", that decomposes an input text prompt into a set of interpretable elements. This decomposition is achieved by learning a pseudo-token, which is a sparse weighted combination of tokens from the model's vocabulary. The goal is to reconstruct the images generated for a given concept using this pseudo-token. The method facilitates single-image decomposition into tokens and semantic image manipulation.", "questions": "See the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors propose a novel view to interpret the internal representations of T2I diffusion model, decomposing the input text prompts into a set of prototypes. Image manipulation can be implemented by simply adjusting the coefficients of these prototypes  \n\nThe method is general and flexible, as it can be applied to any T2I diffusion model without modifying the model architecture or training procedure. \n\nThe writing is good and clear. The paper also provides empirical evidence to support the effectiveness and efficiency of the method.", "weaknesses": "The number of concepts (prototype) are limited, which can not prove whether the proposed method is effective on large-scale concepts.\n\nThe concept decomposing can be viewed as an inner interpolation between the concepts. What if the image are out of domain? Is it possible to show some cases? Can you provided some analysis the between the proposed method and interpolation method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w3NGHUWZ5S", "reviewer": "Reviewer_3WNH", "summary": "This paper attempts to explore the correlations between different textual concepts, by exploring how well they can help reconstruct images of a certain concept with diffusion models. The method is a variation of textual inversion, by incorporating many words from a vocabulary and learning the weights of words (instead of embeddings a new word).", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of finding the correlations between different textual concepts is interesting. The authors presented some interesting results, such as \"snake = twisted + gecko\" (Figure 2).", "weaknesses": "** EDIT** Thanks the author response. I've verified and indeed in the CLIP text space, the triangular similarity relationships are very noisy and seem not reflect true semantic similarity. Therefore I'd raise my rating to 6. As a relevant piece of observation, my original comments on the CLIP triangular relationships are kept as below.\n\n====== original comment ======\n\nMy biggest concern is that it's unnecessary to use image reconstruction as a proxy to find the combination weights (Eq.3). This method can totally work in the CLIP text space only. I've tried to compute the textual similarity of the words presented in Figure 2:\n\nTriplet: 'camel' and 'giraffe'  'cashmere'\n- 'camel' vs 'giraffe': 0.834\n- 'camel' vs 'cashmere': 0.774\n- 'camel' vs 'giraffe' + 'cashmere': 0.872\n\nTriplet: 'snail' and 'ladybug'  'winding'\n- 'snail' vs 'ladybug': 0.768\n- 'snail' vs 'winding': 0.816\n- 'snail' vs 'ladybug' + 'winding': 0.855\n\nTriplet: 'dietitian' and 'pharmacist'   'nutritious'\n- 'dietitian' vs 'pharmacist': 0.878\n- 'dietitian' vs 'nutritious': 0.874\n- 'dietitian' vs 'pharmacist' + 'nutritious': 0.915\n\nTriplet: 'snake' and 'twisted'  'gecko'\n- 'snake' vs 'twisted': 0.869\n- 'snake' vs 'gecko': 0.848\n- 'snake' vs 'twisted' + 'gecko': 0.913\n\nTriplet: 'reflections of earth' and 'sphere'    'civilization'\n- 'reflections of earth' vs 'sphere': 0.761\n- 'reflections of earth' vs 'civilization': 0.804\n- 'reflections of earth' vs 'sphere' + 'civilization': 0.831\n\nTriplet: 'fear' and 'scream'    'wolf'\n- 'fear' vs 'scream': 0.892\n- 'fear' vs 'wolf': 0.875\n- 'fear' vs 'scream' + 'wolf': 0.926\n\nWe can see that for a triplet A,B,C, the similarity of A vs. (B+C) is always higher than A vs B or A vs C. That means similar semantic correlations already exist in the CLIP text embedding space. Intuitively, since CLIP text embeddings are to be aligned with image features, such similarities in the image features will propagate to the text embedding space.\n\nTherefore, doing image reconstruction with T2I diffusion model is unnecessary. If we only mine such triplets from the CLIP text embedding space, then the contribution of this paper becomes quite small. Therefore, I suggest rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AtY8QAgNS1", "reviewer": "Reviewer_nkKA", "summary": "The paper ‘Hidden Language of Diffusion Models’ designs an interpretability framework for text-to-image generative models. This framework relies on learning coefficients of word-embeddings such that the reconstruction loss in diffusion models is minimized with additional constraints to ensure sparsity of concepts which are selected.  Overall, the paper provides a simple framework to decompose concepts into sub-concepts to interpret diffusion models.", "questions": "Refer to the previous section. \n\nOverall, I feel that the paper is good but will like the authors to respond to the Cons/Questions.  The major question I have about this framework,(i)  is how can it be used to mitigate some of the issues in diffusion models (e.g., bias)?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is extremely well-written and easy to follow throughout. Good job on this!\n- The idea about learning the coefficients of the word-embeddings while minimizing the diffusion reconstruction loss is very simple and easy-to-implement while producing good interpretability understanding. It can be a good tool to interpret diffusion models through the lens of how concept dissection works in text-to-image generative models. \n- Given that there does not exist significantly good benchmarks on testing concept-decomposition and also baselines, the authors have done a satisfactory job of comparing with PEZ and other heuristic variants using BLIP-2. The ablations are also presented in depth.", "weaknesses": "Cons / Questions : \n- While the paper provides a good interpretability framework for image generation through the lens of concepts I have some doubts on it’s downstream application. Can the authors elaborate a little bit on how Conceptor can be used for a particular downstream application (e.g., bias mitigation given that Conceptor can detect biases?)\n- Can the authors elaborate if the concept decomposition is an artifact of the particular CLIP text-encoder in Stable-Diffusion? Will one get similar concept-decomposition patterns if a different text-encoder is used (e.g., T5 like in DeepFloyd)? I would imagine this to be a positive answer, but might expect different patterns, so I believe it’s important to use Conceptor to understand this phenomenon.\n- I am a little curious about how much the diffusion objective plays a role in concept-decomposition. For e.g., given the objective of reconstruction, I will expect the faithfulness metric of Conceptor to be better than other methods (e.g., PEZ). However, if you use the same idea with CLIP loss (replacing the reconstruction loss in Eq. (6) with L_clip), will you get similar decomposition? And will those decompositions transfer to diffusion models?  In fact, if you use CLIP's representation for a particular token as a ground-truth with optimizing for Eq.(3), you should get a reasonable reconstruction still, which can be a cheap baseline. Did the authors run this ablation?\n- How can you extend your framework to more complex concept-decomposition? The current framework generates images corresponding to single concepts, but images are usually consisting of multiple concepts. In this scenario, how can one use Conceptor to understand sub-concepts? I think this is one experiment, the paper is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJpLv8KHOw", "reviewer": "Reviewer_vrPv", "summary": "The paper proposes to use a weighted linear combination of existing word embeddings to represent an image (i.e., weights are optimized through two MLP layers) such that it enables image decomposition based on a set of human understandable tokens using pretrained text-to-image diffusion model such as Stable diffusion models.", "questions": "- I'd love to see if u can optimize on electrician image and then try removing the brush concept, does it become a painter? \n- How long does this optimization process take for each image?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- the motivation of the paper is clear and well conveyed.\n- the experiments are well-conducted and extensive.\n- The approach enables human interpretable decomposition using a set of learnable weights and associated tokens, which has inherited the same outcome from the classic word2vec arithmetic paper [1].\n\n[1] Mikolov et al., Efficient Estimation of Word Representations in Vector Space. ICLR 2013", "weaknesses": "- lack of related previous work on image decomposition, which can be seen as a way to interpret models, such as FineGAN (Sing et al, 2019), GIRAFFE (Niemeyer etal, 2021), SlotAttention (Locatello etal, 2020), DTI Sprites (Monnier etal, 2021), GENESIS-V2 (Engelcke etal, 2021) and follow-up works. There also exists earlier/concurrent work that conducts image decomposition using text-diffusion models, so its worth discussing pros and cons but may not need comparison if works are concurrent.\n- most of concepts shown in the paper are mainly objects, thus its ability to learn abstract concepts is not clear. For example, how does it perform on abstract concepts such as object relationships. Though, I tend to think that the model is rather limited in understanding complex concepts other than objects.\n- The method essentially utilizes arithmetic with word embeddings which have been widely used in the past, so it doesn't seem to be novel enough. Applying this method to a text-to-image diffusion model doesn't show novelty from my perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)