Emergence and Evolution of Interpretable Concepts in Diffusion Models
Authors: Berk Tinaz
Zalan Fabian
Mahdi Soltanolkotabi
Venue: NeurIPS 2025 spotlight ()
Keywords: mechanistic interpretability
diffusion models
sparse autoencoders
controlled generation
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We investigate how human-interpretable concepts evolve in diffusion models through the generative process.

Abstract: Diffusion models have become the go-to method for text-to-image generation, producing high-quality images from pure noise. However, the inner workings of diffusion models is still largely a mystery due to their black-box nature and complex, multi-step generation process. Mechanistic interpretability techniques, such as Sparse Autoencoders (SAEs), have been successful in understanding and steering the behavior of large language models at scale. However, the great potential of SAEs has not yet been applied toward gaining insight into the intricate generative process of diffusion models. In this work, we leverage the SAE framework to probe the inner workings of a popular text-to-image diffusion model, and uncover a variety of human-interpretable concepts in its activations. Interestingly, we find that *even before the first reverse diffusion step* is completed, the final composition of the scene can be predicted surprisingly well by looking at the spatial distribution of activated concepts. Moreover, going beyond correlational analysis, we design intervention techniques aimed at manipulating image composition and style, and demonstrate that (1) in early stages of diffusion image composition can be effectively controlled, (2) in the middle stages image composition is finalized, however stylistic interventions are effective, and (3) in the final stages only minor textural details are subject to change.

Emergence and Evolution of Interpretable Concepts in Diffusion Models
Berk Tinaz' Zalan Fabian" Mahdi Soltanolkotabi
Dept. of Electrical and Computer Engineering University Southern California Los Angeles, CA, USA fabian. www.anamaillaaacmmmmmmmmmm com soltanol cesoollusc.eee
tinazBusc. edu
Abstract
Diffusion models have become the go-to method for text-to-image generation, producing high-quality images from pure noise. However, the inner workings diffusion models is still largely a mystery due their black-box nature and com- plex, multi-step generation process. Mechanistic interpretability techniques, such Sparse Autoencoders (SAEs), have been successful in understanding and steer- ing the behavior of large language models at scale. However, the great potential of SAEs has not yet been applied toward gaining insight into the intricate generative process of diffusion models. In this work, we leverage the SAE framework to probe the inner workings of a popular text-to-image diffusion model, and uncover a variety of human- suman-interpreeable concepts in its activations. Interestingly, we find that even before the first reverse diffusion step is completed, the final composition of the scene can be predicted surprisingly well by looking at the spatial distribution activated concepts. Moreover, going beyond correlational analysis, we design intervention techniques aimed at manipulating image composition and style, and demonstrate that (1) early stages of diffusion image composition can effec- tively controlled, (2) in the middle stages image composition is finalized, however stylistic interventions are effective, and (3) the final stages only minor textural details are subject to change."
Introduction
Diffusion models (DMs) 17 47 have revolutionized the field of generative modeling. These models
iteratively refine images through a denoising process, progressively transforming Gaussian noise into coherent visual outputs. DMs have established state-of-the-art in image [8] 33, 42
40) [8), audio 24]. and video generation 19]. The introduction of text-conditioning diffusion models
i.e. guiding the generation process via text prompts, enables careful customization of generated samples while simultaneously maintaining exceptional sample quality.
While DMs excel at producing images of exceptional quality, the internal mechanisms by which they ground textual concepts in visual features that govern generation remain opaque. The time- evolution of internal representations through the generative process, from pure noise to high-qualig images, renders the understanding of DMs even more challenging compared other deep learning models. A particular blind spot is the early, 'chaotic' stage [53] of diffusion, where noise dominates the generative process. Recently, flurry of research has emerged towards demystifying the inner workings of DMs. In particular, line of work attempts to interpret the internal representations by constructing saliency maps from cross-attention layers [50] Another direction find interpretable
"quual contribution.
Coode is available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

editing directions directly in the model's 's feature space that allows for guiding the generation process However. most existing techniques are aimed at addressing particular editing tasks and are not wide enough in scope to provide a more holistic interpretation on the internal representations of diffusion models.
[5
14
36
5 35
9
Mechanistic interpretability (MI) (34] is focused on addressing the above challenges via uncovering operating principles from inputs outputs that reveal how neural networks process information internally. A line of work within MI uses linear or logistic regression on model activations, also known as probing 13
to uncover specific knowledge stored in model internals. Extensions 20 I explore nonlinear variants for improved detection and model steering. Recently, sparse autoencoders have emerged within MI powerful tools to discover highly interpretable features (or concepts) within large models at scale [6]. These learned features enable direct interventions to steer model behavior in a controlled manner. Despite their success understanding language models, the application of SAEs to diffusion models remains largely unexplored. Recent work 449
leverages While the
SAEs and discovers highly interpretable concepts in the activations of distilled DM 43
results are promising, the paper focuses on single-step diffusion model, and thus the time-evolution of visual features, a key characteristic and major source of intrigue around the inner workings of DMs, is not captured in this work.
In this paper, we aim bridge this gap and address the following key questions:
What level of image representation is present in the early stage of the generative process?
How do visual representations evolve through various stages of the generative process?
Can we harness the uncovered concepts to steer the generative process in an interpretable way?
How does the effectiveness of such interventions depend on diffusion time?
We perform extensive experiments on the features of a popular, large-scale text-to text-to-image DM, Stable Diffusion v1.4 [40], and extract thousands of concepts via SAEs. We propose novel, scalable, vision- only pipeline to assign interpretations to SAE concepts. Then, we leverage the discovered concepts to explore the evolution of visual representations throughout the diffusion process. Strikingly, we find that the coarse composition of the image emerges even before the first reverse diffusion update step, which stage the model output carries identifiable visual information (see TT. Moreover, we demonstrate that intervening on the discovered concepts has interpretable, causal effect on the generated output image. We design intervention techniques that edit representations in the latent space of SABs aimed at manipulating image composition and style. We perform an in-depth study on the effectiveness of such interventions as reverse diffusion progresses. We find that image composition can be effectively controlled in early stages of diffusion, however such interventions are ineffective in later stages. Moreover, we can manipulate image style at middle time steps without altering image composition. Our work deepens our understanding on the evolution of visual representations in text-to-image DMs and opens the door to powerful, aime-adaptive editing techniques.
Background
Diffusion models- In the diffusion framework, forward noising process progressively transforms the clean data distribution 2g qu (x:) into a simple distribution 4r (typically isotropic Gaussian distribution) through intermediary distributions qt In general, gt is chosen such that æ; obtained
by mixing zy with an appropriately scaled i.i.d. Gaussian noise, (æ/(=z) ~ - (Eo. ofI) where
the variance of is chosen according to a variance schedule. Diffusion models [46] 17 47 48 learn to
reverse the forward process to generate new samples from qo by simply sampling from the tractable distribution ęr. Throughout this paper, we assume that the diffusion process is parameterized by a continuous variable E [0, 1]. where = 1 corresponds to pure noise distribution and t corresponds to the distribution of clean images.
Sparse autoencoders (SAEs) - Sparse autoencoders are one of the most popular mechanistic interpretability techniques, and have been demonstrated to find interpretable features at scale [6] 12] The core assumption underpinning SAES is the superposition hypothesis, the idea that models encode far more concepts than the available dimensions in their activation space by using combination of sparse and linear representations 45) SAES unpack these features in an over-complete basis of sparsely activated concepts in their latent space, as opposed to the compressed latent space of
2

---

Several walking the dirt with palm trees in the background.
Generated image
Predicted layout Diffusion output
First generation step
Figure I: General scene layout emerges during the very first generation step in diffusion models. We generate an image with the prompt Several men walking on the dirt with palm trees in the background. Our interpretability framework object mentioned in the input prompt, solely relying on model activations cached during the first diffusion step. At this early stage, the posterior mean predicted by the diffusion model does not contain any visual clues about the final generated image.
predict segmentation masks for each
autoencoders commonly used in representation learning. Training autoencoders with both low reconstruction error and sparsely activated latents is not an easy feat. An initial approach [2] towards this goal uses ReLU as the activation function and & loss as regularizer to induce sparsity. However, additional tricks are necessary, such as the initialization of encoder and decoder weights, to ensure that training is stable. Moreover, auxiliary loss terms may be necessary to ensure there are no dead neurons/concepts. Recent work 31 3 proposes using TopK activation instead of the ReLU function, which enables the precise control of the sparsity level without ε loss and results improved downstream task performance over ReLU baselines.
Interpreting diffusion models- There has been significant effort towards interpreting diffusion models. Tang et al. [50] find that the cross-attention layers in diffusion models with a U-Net backbone -such as SDXL 38 and Stable Diffusion 40] can be used generate saliency maps corresponding to textual concepts. Another line of work focuses on finding interpretable editing directions in diffusion U-Nets control the image generation process. For instance, Kwon et al. [25] and Haas et al. 14 manipulate bottleneck features, Park et al. [6)] identifies edit directions based on the SVD of the Jacobian between the input and bottleneck layer of the U-Net, while Chen et al. 5 considers the Jacobian between the input and the posterior mean estimate. Other works modify the key and value projection matrices 35 or directly control object attributes by thresholding attention maps 99 4
Beyond editing directions, prior studies have noted the emergence of coarse image structure in the early diffusion steps [27 [5] and the evolution of high- and low-level semantics through different contrast, our SAE-based analysis provides complementary yet more granular and systematic characterization of these phenomena; simultaneously enabling interpretable editing and semantic layout prediction through concept vectors discovered by SAEs.
stages of the diffusion process 37 30
Recent work 44 trains SAES on activations of a distilled, single-step diffusion model (SDXL Turbo) 43
Authors target residual updates in specific cross-attention blocks of the U-Net and found the features in the latent space of SAFS are to be highly interpretable. Our approach differs in two key ways. First, we analyze the time-evolution of interpretable concepts during generation critical for understanding and controlling diffusion which single-step models cannot capture. Second, they rely on vasion-language foundation models to interpret SAE features by summarizing commonalities among set of images that activate specific features. Alternatively, we introduce a scalable pipeline that builds a flexible concept dictionary that further supplies us with pixel-Ievel annotations using open-set object detectors and segmentation models. Concurrently, Cywinski and Deja M7] apply SAES to machine unlearning in non-distilled diffusion models. However, they focus on concept removal and train a single SAE across all time steps, whereas we study temporal dynamics by training time-specific SAES. Lastly, Kim and Ghadiyaram (22 use SAFS for controlled generation, but operate text-encoder activations rather than visual representations within the diffusion model.

---

3 Method
3.1 SAE Architecture and Loss
In this section, we discuss the design choices behind our SAE model. We opt for k-sparse au- toencoders (with TopK activation) given their success with GPT-4 [2] and SDXL Turbo 49]. In particular, let æ: E Rd denote the input activation to the autoencoder that we want to decompose into sparse combination of features. Then, we obtain the latent E R by encoding a as = E (a+) = Topk (ReLU (Wene (æc b)]). where W enc E R denotes the learnable weights of the encoder, b E IR is learnable bias term, and TopK function keeps the top k highest activations and sets the remaining ones to O. Note, that due to the superposition hypothesis, we wish the encoding to be expansive and therefore n > d. Then, decoder is trained to reconstruct the input from the latent in the form å = D (z) = Wdec + b. where Wdec Rdx,/ represents the learnable weights of the decoder. Note, that the bias term is shared between the encoder and decoder. We refer to f: = Wdeel:, columns of Wdec as concept vectors. We discuss additional details on SAE training Appendix A
3.2 Collecting Model Activations
In this work, we use Stable Diffusion v1.4 (SDv1.4) 40 as our diffusion model due to its widespread use. Inspired by Surkov et al. [49], we use 200k training prompts from the LAION-COCO dataset 44 and store de, E x the difference between the output and input of the fth cross- attention transformer block at diffusion time t (i.e. the update to the residual stream). We train our SAE to reconstruct features individually along the spatial dimension. That the input to the SAE is (e, j for different spatial locations (i,j) whereas E and are fixed and to be specified next.
To capture the time-evolution of concepts, at timesteps corresponding to I E [0.0, 0.5, 1.0] and analyze final (t = 0.0, close to fi- nal generated image), middle, and early (l. = 1.0, close to pure noise) diffusion dynam- ies respectively. For each timestep t, we target 3 different cross-attention blocks in the denoising model of SDv1.4: down_blocks
collect activations across 50 DDIM steps
attentions..... mid_block attentions..... .0, up_ blockl...11111... 1. attentiinns ons O. We refer to these as down_block mid_block up_ block for brevity. We specifically include the mid_mllddlllle blockboockkooooo or the bottleneck layer of the U-Net since earlier work found interpretable editing directions here (25). Other blocks are chosen to be the closest to the bottleneck layer in the downsampling and upsampling paths of the U-Net. The performance of text guidance is improved through Classifier-Free Guidance (CFG) [16]. The model output is modified as EØ ( L, c) = Eg (Ze,4 l, c) + w (Eg( e, E L, Ø)), where w denotes the guidance scale, is the conditioning input and Ø is the null-text prompt. At each timestep we collect both the text-conditioned diffusion features (called cond) and null-text-conditioned features (denoted by uncond).
To provide an in-depth analysis, we train separate SAEs for different block, conditi tioning and t timestep combinations. Training results are in Appendix [ In this work, we focus on cond features, as we hypothesize that they may be more aligned with human- -nterrrttteeeeebalee concepts due the direct influence of language guidance through cross-attention (more on this in Appendix C).
3.3 Extracting Interpretations from SAE Features
Multiple work on automatic labeling of SAR features resort to LLM pipelines where the captions corresponding top activating dataset examples are collected and the LLM is prompted summarize them. However, these approaches come with severe shortcomings. First, they may incorporate the biases and limitations of the language model into the concept labels, including failures in spatial reasoning [21] object counting, identifying structural characteristics and appearance (51) and object hallucinations (26). Second, they are sensitive the prompt format and phrasing. and the instructions may bias or limit the extracted concept labels. Last but not least, it is computationally infeasible to scale LLM-basee concept summarization to a large number of images, limiting the reliability of extracted concepts. For instance, Surkov et al. 49 only leverages a few dozens of images to define each concept. Therefore, we opt for designing scalable approach that obviates the need for LLM-based labeling and instead use vision-based pipeline to label our extracted SAE features.

---

In particular, we represent each concept by an associated list of objects, constituting a concepr dictionary. The keys are unique concept identifiers (CIDs) assigned to each of the concept vectors of the SAE. The values correspond to objects that commonly occur in areas where the concept is activated. To build the concept dictionary, we first sample set of text prompts, generate the corresponding images using diffusion model and extract the SAE activations for each CID during generation. We obtain ground truth annotations for each generated image using a pre-trained vision pipeline, that combines image tagging, object detection and semantic segmentation, resulting in mask and label for each object in generated images. Finally, we evaluate the alignment between our ground truth masks and the SAE activations for each CID, and assign the corresponding label to the CID only if there is sufficient overlap. We refer the reader to Appendix F.I for a detailed depiction of the pipeline.
The concept dictionary represents each concept with a list of objects. In order to provide a more concise summary that incorporates semantic information, we assign an embedding vector to each concept. In general, we could use any model that provides robust natural language embeddings, such as LLM, however we opt for a simple approach by assigning the mean Word2 embedding of object names activating the given concept.
3.4 Predicting Image Composition from SAE Features
Leveraging the concept dictionary, predict the final image composition based on SAE features any time step, allowing us to gain invaluable insight into the evolution of image representations in diffusion models. Suppose that would like predict the location of a particular object in the final generated image, but before the reverse diffusion process is completed. First, given SAE features from given intermediate time step, we extract the top activating concepts for each spatial location. Next, we create conceptual map of the image by assigning a word embedding to each spatial location based on our curated concept dictionary. This conceptual map shows how image semantics, described by localized word embeddings, vary spatially across the image. Given a concept we would like to localize, such as an object from the input prompt, we produce target word embedding and compare its similarity to each spatial location in the conceptual map. To produce a predicted segmentation map, we assign the target concept to spatial locations with high similarity, based on a pre-defined threshold value. This technique can be applied to each object present in the input prompt (or to any concepts of interest) to predict the composition of the final generated image. We provide detailed visualization of our technique in Appendix F22
3.5 Causal Intervention Techniques
Analyzing top activating dataset examples and semantic segmentation predictions only establish correlational relationship between concepts and the output image. In order to probe causal effects, we consider two categories of interventions: spatially targeted interventions designed to guide scene layout and global interventions directed towards manipulating image style.
Spatially targeted interventions - To assess layout controllability using the discovered concepts, we propose a simple task: enforce specific object to appear only in a designated quadrant (e.g., top-left) of the image. To achieve this, we intercept activations and edit features in the SAE latent space by amplifying the desired concept in the target region and setting it to 0 otherwise. Recall, that the contribution of the 9th transformer block at time t is given by Dr,-- Let Ze, denote the latents after encoding the activations with the SAE encoder E. Let S denote the set of coordinates to which we would like to restrict the object. Let C. be the set of CIDs that are relevant to object o. We wish to modifty the latents as follows:
B, if (i,))
VC Co. Ze. =
(I)
00 otherwise
where 3 is our intervention strength. However, decoding the modified latents directly is suboptimal as the SAE cannot reconstruct the input perfectly. Instead, we modify the activations directly using the concept vectors. The modification in Eq. can be equivalently written as:
Å Ae.li.jl = AA.,ii, (i ++ Sa if (1,))
(2)
A Eeco, Scr otherwise
5

---

Prompt
"Ao apple basket.
Latent Space
Pixol
Dansiiing Litty
Q
0
Intervention
"Concepte"
Goal
toblle
Activatins
Mediiiee
An overview of our SAE intervention technique. The prompt "An apple in a basket" specifies the necessary concepts but is vague terms of spatial composition. We intercept activations of the denoising model and edit the latents after encoding them with the SAE. For the features that are spatially located in the bottom-right quadrant, we increase the coefficient corresponding to "apple" concept, while setting it for all other features. After the intervention, generated image satisfies the specified layout where the apples are located in the Nottom-right quadrant.
Figure 2:
An overview of this intervention can be in Figure 2 In prior experiments, observe that the same intervention strength 3 does not work well across different objects o. To solve this, we introduce normalization where the intervention at spatial coordinate (i, is proportional the norm of the latent at that coordinate [i, Therefore, the effective intervention strength is Bi 3 J]]l
Global interventions Beyond image composition, we investigate whether image style can be manipulated through discovered concepts. To this end, given CID related to the style of interest, as image style a global property we modify the activation at each spatial location as
Á j] Ae. + 3fc-
Similar to spatially targeted interventions, find that normalization is necessary for work well across different choices of style. We let (3 to be adaptive to spatial locations and modify them as
B. Σ., [5 3.
[R.5]
4
Experiments
We perform extensive experiments on SD v1.4 aimed at understanding how internal representations emerge and evolve through the generative process.
4.1 Building the Concept Dictionary
We sample 40k prompts from the LAION-COCO dataset from a split that has not been used to train the SAEs. We build the concept dictionary following our technique introduced in Section 3.3
For annotating generated images, we leverage RAM [54) for image tagging. Grounding DINO 28
for open-set object detection and SAM (23] for segmentation, following the pipeline in Ren al.
39 We assign label to specific CID if the loU between the corresponding annotated mask and activation is greater than 0.5. We binarize the activation map for the loU calculation by first normalizing to [0, 1] range, then thresholding at 0.1. We visualize the top activating concepts and the corresponding concept dictionary entries in Appendix F3
4.2 Emergence of Image Composition
Next, we investigate how image composition emerges and evolves in the internal representations of the diffusion model. We sample 5k LAION-COCO test prompts that have not been used for SAE training or to build the concept dictionary, and generate corresponding images with SDv1.4. Then,

---

Goscale image
05
0.34
0.32
loU
0.30
0.28
0.26
1.0
0.5
0.0
Diffusion time
(a) Evolution of predicted image composition accuracy (in terms of IoU) over the reverse diffusion process
(b) Visualization of segmentation maps predicted from extracted concepts across reverse diffusion steps (up_block).
Figure 3: Evolution of predicted image composition during the reverse diffusion process, shown through segmentation accuracy (left) and visualizations (right). Features from later time steps become progressively more accurate at predicting the final layout of the image. However, the general image composition emerges as early as the first time step.
we follow the methodology described in Section 3.4 to predict segmentation mask for every noun in the input prompt using SAE features at various stages of diffusion. We filter out nouns that are not in Werd2Vec and those not detected in the generated image by our zero-shot labeling pipeline. We evaluate the mean IoU between the predicted masks and the ground truth annotations from our labeling pipeline for the first generation step (t = 1.0), the middle step (t = 0.5) and final diffusion step 0.0). Numerical results are summarized in Figure 3a
First, we surprisingly find that the image composition emerges during the very first reverse diffusion step (even before the first complete forward pass!), we are able predict the rough layout of the final scene with LoU 2 0.26 from mid_ block SAE activations. As | demonstrates,, the general location of objects from the input prompt is already determined at this stage, even though the model output (posterior mean prediction) does not contain any visual clues about the final generated scene yet. More examples can be seen in the second column of Figure 3b
Second, we observe that the image composition and layout is mostly finalized by the middle of the reverse diffusion process (1 0.5), which is supported by the saturation in the accuracy of predicted masks. Visually, predicted masks for f = 0.5 and / = 0.0 look similar, however we see indications of increasing semantic granularity in represented concepts. For instance, the second row in Figure 3b]depicts predicted segmentation masks for the noun church. Even though the masks for = 0.5 and 0.0 are overall similar, the mask in the final time step excludes doors and windows on the building, suggesting that those regions are assigned more specific concepts, such as door and window. Moreover, we would like emphasize that the segmentation lo is evaluated with respect to our zero-shot annotations, which are often less accurate than our predicted masks for t = 0.0, and thus the reported IoU is bottlenecked by the quality of our annotations.
Finally, we find that image composition can be extracted from any of the investigated blocks, and thus we do not observe strong specialization between these layers for composition-rrlated information. However, up_ sp_block provides generally more accurate segmentations than down down_blowe block, and mid_ block provides the lowest due the lower spatial resolution. We also find that cond features result in more accurate prediction of image composition than uncond features, likely due more semantic information as an indirect result of text conditioning. Results for all block and conditioning combinations can be found in Appendix
4.3 Effectiveness of Interventions Across Diffusion Time
Beyond establishing correlational effects, we analyze how our discovered concepts can be leveraged in causal interventions targeted manipulating image composition and style. We specifically focus on the effectiveness of these interventions function of diffusion time, split into 3 stages: early

---

(a) Etrly-stage intervention.
(b) Mtddle-stage intervention
(c) stage intervention
Figure 4: Effect of spatially targeted interventions different stages of diffusion, aimed manipulat- ing image layout. We can restrict objects to the specified quadrant of the image when intervened in early stages of diffusion. However, in middle and final stages our interventions unsuccessful.
for 1 E [0.6, 1.0), middle for E [0.2, 0.6) and final for & E [0,0.2]. Motivated by the success of bottleneck intervention techniques [25 14 36), we target mid_ in experiments.
Spatially targeted interventions- We consider bee, book, and dog as the objects of interest and at- tempt to restrict them to four quadrants: top-left, top-right, bottom-left, and bottom-rige In order to find the CIDs to be intervened on, we sweep the concept dictionary of the given time step and collect all the CIDs where the word of interest appears. Results are summarized in Figure
Global interventions- Through our concept dictionary and visual inspection of top dataset examples at = 0.5, we select the following CIDs: #1722 that controls the cartoon look of the image, #524 appears mostly with beach images where and sand are visible together, and #2137 activates the most on paintings (top activating images can be found in Appendix [H). We find matching concepts for other time steps by picking the CIDs with the highest Word2Vec embedding similarity to the above target CIDs, An overview of results is depicted in Figure 5
4.3.1 Etrly-stage Interventions
First, we apply spatially targeted interventions according to Eq. using an SAE trained on cond activations of mid_block at t = 1.0. We observe that large intervention strength / is needed to successfully control the spatial composition consistently. We hypothesize that the skip connections in the U-Net architecture and the features from the null-tex: conditioning in classifier-free guidance reduce the effect of our interventions, as they provide paths that bypass the intervention. Thus, a larger value of intervention strength is needed to mask the leakage effects. In Figure 4aj we observe that the objects of interest are successfully guided their respective locations. Moreover, the concepts that we do not intervene on, such as the flower in the first row are preserved.
Next, we perform global interventions according to Eq. aimed at manipulating image style. Interestingly, as depicted in Figure 5a we find that instead of controlling image style, these global interventions broadly modify the composition of the image. without imbuing it with a particular style. As depicted in Figure 6 (top row), this phenomenon holds for a wide range of 3. As we vary the intervention strength, we obtain images with various compositions, but without the target style. This observation is consistent with our hypothesis that early stages of diffusion are responsible for shaping the image composition, whereas more abstract and high-level concepts, such as those related to consistent artistic styles emerge later.
4.3.2 Middle-stage Interventions
We keep the setting from early-stage experiments, but use an SAE trained on the activations at t 0.5. In contrast with early-stage results, as shown in Figure 4b) we find that our spatially localized intervention fails to manipulate image composition at this stage. This result suggests that the locations of prominent objects in the scene have been finalized by this stage. The interventions cause visual distortions, while maintaining image composition. Interestingly, in some cases we see semantic changes in the targeted regions. For instance, intervening on the book concept in the second row of Fig. (4b] in the top-left quadrant changes the tea cup into a book, instead of moving the large book making up most of the scene.
8

---

No Intvmaii
CRESAOS
reC aand
Car
oast
(a) Etaly-stage intervention
(b) Middle-stage intervention
(c) Final-stage intervention
Figure 5: Effect of global interventions aimed at manipulating image style. Intervening in the early stages of diffusion drastically modifies image composition without imbuing the image with particular style. In stark contrast, middle-stage global interventions successfully manipulate image style without interfering with image composition. However, in the final stages of diffusion, such global interventions have no effect on style or composition, and only result in minor textural changes.
In an effort to control image style, we perform global interventions in the middle stages. We show results in Figure 5b We find that the these interventions do not alter image composition as in early stages of diffusion. Instead, we observe local edits more aligned with stylistic changes (cartoon look, sandy texture, smooth straight lines, etc.), while the location of objects in the scene are preserved. Contrasting this with early-stage interventions, diffusion is responsible for the emergence of more high-level and abstract concepts whereas the image layout is already determined in the carlier time steps (also supported by our semantic segmentation experiments). Moreover, varying the intervention strength impacts the intensity of style transfer in the output image (middlle row)).
hypothesize that the middle stage of
4.3.3 Final-stage Interventions
Performing spatially targeted interventions in the final stage of diffusion (Figure 4c) has no effect on image composition and only causes some minor changes in local details. This outcome is expected, as we observe that even by the middle stages of diffusion, image composition is finalized.
Similarly, we find that our global intervention technique is ineffective in manipulating image style in the final stage of diffusion (Figure as we only observe minor textural changes across wide range of intervention strengths (Figureld (bottom row)).
4.4 Summary of observations
Our experimental observations can be summarized as follows:
Early stage of diffusion: coarse image composition emerges as early as during the very first diffusion step. At this stage, are able to approximately identify where prominent objects will be placed in the final generated image (Section 4.2 and Figure []. Moreover, image composition is still subject change: we manipulate the generated scene (Figure 4a) by spatially targeted interven- tions that amplify the desired concept in some regions and dampens it in others. However, we are unable to steer image style (Figure 5a this stage using our global intervention technique. Instead of high-level stylistic edits, these interventions result in major changes in image composition.
Middle stage of diffusion: image composition has been finalized at this stage and we are able to predict the location of various objects in the final generated image with high accuracy (Figure 3). Moreover, our spatially targeted intervention technique fails to meaningfully change image composition at this stage (Figure 4b). On the other hand, through global interventions we can effectively control image style (Figure 5bp while preserving image composition, in stark contrast to the carly stages.

---

8:00
10
#:400
8 100
Eudyy-tuge
Mkddle
Pralstige
Figure 6: Effecc of intervention strength. We perform global intervention on concept (# 1722 for all time steps) corresponding to cartoon look in top activating images. Early-stage interventions, at any strength, unable to modify image style consistently but broadly influence image composition. Interventions in the middle stages imbue the image with the target style with increasing intensity. We only observe minor textural changes in final stages of diffusion, even at high intervention strengths.
Final stage of diffusion: Image composition can be predicted from internal representations to very high accuracy (empirically, often higher than pre-trained segmentation pipeline), however manipulating image composition through our spatially localized interventions fail (Figure 4c). Our global intervention technique only results in minor textural changes without meaningfully changing image style (Figure (5c). These observations are consistent with prior work [53] highlighting the inefficiency of editing the final, 'refinement" stage of diffusion.
While these observations are based on the SDv1.4 diffusion model, we expect similar behavior from different models and architectures. In partiular. we hypothesize that the concepts that are learned by the model are mostly due to the diffusion objective rather than the precise choice of denoising architecture. As noted by Fuest al. [0]. the denoising objective in diffusion models encourages the learning of semantic image representations useful for downstream tasks, much like Denoising Autoencoders (DAEs). The key distinction is that diffusion models condition on the timestep t, effectively functioning hierarchy of DAEs operating multiple noise levels. Building on this close analogy between DAES and diffusion models (rigorously analyzed in (52]), we believe that even if the denoising model architecture changes (such as DiT used in FLUX models vs. the U-Net in SDv1.4, SDXL, etc.), similar underlying concepts should emerge when probed appropriately due to the fundamentally related representations learned by diffusion models.
We also include more quantitative experiments demonstrating that concepts become more refined and distinct as generation progresses (Appendix [E.I). and assessing the success of global and spatially targeted interventions at different diffusion stages which conforms with our summary of observations in this section (Appendix E.2).
5 Conclusions and Limitations
In this paper, we take a step towards demystifying the inner workings of text-to-image diffusion models under the lens of mechanistic interpretability, with an emphasis on understanding how visual representations evolve over the generative process. We show that the semantic layout of the image emerges as early as the first reverse diffusion step and can be predicted surprisingly well from our learned features, even though no coherent visual cues are discernible in the model outputs this stage yet. As reverse diffuusion progresses, the decoded semantic layout becomes progressively more refined, and the image composition largely finalized by the middle of the reverse trajectory. Furthermore, we conduct in-depth intervention experiments and demonstrate that we can effectively leverage the learned SAE features to control image composition in the early stages and image style in the middle stages of diffusion. Developing editing techniques that adapt to the evolving nature of diffusion representations is promising direction for future work. A limitation of our method is the leakage effect rooted in the U-Net architecture of the denoiser, which enables information to bypass our interventions through skip connections. We believe that extending our work to diffusion transformers would effectively tackle this challenge.
10

---

Acknowledgements
We would like to thank Microsoft for an Accelerating Foundation Models Research grant that provided the OpenAl credits enabling this work. This research is also part supported by AWS credits through an Amazon Faculty research award and a NAIRR Pilot award. Soltanolkotabi also supported by the Packard Fellowship in Science and Engineering, Sloan Research Fellowship in Mathematics, an NSF-CAREER under award #1 1846369, and NSF-CIF awards # #1813877 and #2008443. and NIH DP2LM014564-01.
References
[1] Daniel Beaglchole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, and Mikhail Belkin. Aggregate and conquer: detecting and steering Ilm concepts by combining nonlinear predictors over multiple layers, 2025.
[2] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. hutps:/transformer- circuits.
[3] Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders, 2024.
[4] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3343-5353, 2024.
[5] Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu. Exploring low- dimensional subspaces in diffusion models for controllable image editing, 2024.
[6] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023.
[7] Bartosz Cywiñski and Kamil Deja. Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders. arXiv preprint arXiv:2501.18033 2025.
[8] Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. arXiv preprint vrXiv:2105.0525: 2021.
[9] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation, 2023.
[10] Michael Fuest, Pingchuan Ma, Ming Gui, Johannes Schusterbauer, Vincent Tao Hu, and Bjorn Ommer. Diffusion models and representation learning: survey, 2024.
[11] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyíska, and David Bau. Unified concept editing in diffusion models, 2024.
[12] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024.
[13] Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without generating a single token. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3994 4019, Miami, Florida, USA, 2024. Association for Computational Linguistics.
[14] René Haas, Inbar Huberman-Spien Spaeemaa--piigglllaa, Rotem Mulayoff, Stella Graßhof, Sami S. Brandt, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models, 2024.

---

[15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-On Prompt-to-pronee prottpt image editing with cross attention control, 2022.
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, arXiv preprint arXiv:2006. 100006..1220 11299, 2020.
[18] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
[19] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint FrXXi:2204.03458, 2022.
[20] Jakub Hoscilowicz, Adam Wiacek, Jan Chojnacki, Adam Cieslak, Leszek Michon, Vitalii Ur- banevych, and Artur Janicki. Non-linear inference time intervention: Improving Ilm truthfulness, 2024.
[21] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with vision-language models? investigating their struggle with spatial reasoning. arXiv preprint 1XXi:23310.19883 2023.
[22] Dahye Kim and Deepti Ghadiyaram. Concept steerers: Leveraging k-sparse autoencoders for controllable generations. arXiv preprini arXiv:2501.19066, 2025.
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pages 4015-4026, 2023.
[24] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 2020.
[25] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space, 2023.
[26] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision- langunlanguuge models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13872-13882, 2024.
[27] Shuangqi Li, Hieu Le, Jingyi Xu, and Mathieu Salzmann. All seeds are not equal: Enhancing compositional text-to-image generation with reliable random seeds, 2025.
[28] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. European Conference on Computer Vision, pages 38-55. Springer, 2024.
[29] Timo Lüddeeke and Alexander S. Ecker. Image segmentation using text and image prompts, 2022.
[30] Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for text-to-image diffusion models, 2023.
[31] Alireza Makhzani and Brendan Frey. k-sparse autoencoders, 2014.
[32] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Work- shop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Singapore, 2023. Association for Computational Linguistics.
[33] Alex Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models, arXiv preprint axXiv:2102.09672 2021.
12

---

[34] Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. attps://wwe transformer cir.uirrrrrrtts... puw.l///... int.chh-stttrrrssssaeaee 2022. Accessed: 2025-04-14.
[35] Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to- image diffusion models, 2023.
[36] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space diffusion models through the lens of riemannian geometry, 2023.
[37] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models, 2023.
[38] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[39] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint urXiv:2401.14159, 2024.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image Super- Resolutoso via Iterative Refinement. arXhv:2104.07636 [cs, eess], 2021.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- ii-ageeee diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36444 2022.
[43] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 87-103. Springer, 2024.
[44] Christoph Schuhmann, Andreas Kopf, Richard Vencu, Theo Coombes, Romain Beaumont, and Benjamin Trom. Laion coco: 600m synthetic captions from laion2b-en.
[45] Lee Sharkey, Bilal Chughtai. Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. Open problems in mechanistic interpretability. 2025.
[46] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[47] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. UrXiv:1907.05600 [cs, stat], 2020.
[48] Yang Song and Stefano Ermon. Improved Techniques for Training Score- Based Generative Models. arXiv:2006. 09011 (cs, stat], 2020.
[49] Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoen- coders, 2024.
[50] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention, 2022.
13

---

[51] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2024.
[52] Xingyi Yang and Xinchao Wang. Diffusion model representation learner, 2023.
[53] Jiwen Yu, Yinhuai Wang. Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy gorrgy-guiedd conditional diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23174-23184, 2023.
[54] Youcai Zhang. Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1724 1732, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are supported by extensive experiments in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work are discussed at the end of Conclusions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The full set hyperparameters are provided in the main paper and in the Appendix to train and test the SAEs towards reproducing the main claims in the paper.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We release the code base and accompanying model checkpoints. The data we use is already open-source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
public/guides ottp://seeooooeeebeessssoooooona
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wetgguddddeeouuubbuneeee ssiniooiccy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We share all training and test details in Section 4 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In terms of numerical results, we provide the average performance in the main body of the paper (in Figure Ba and the Appendices and E.2 The numerical results provided are solely to demonstrate trends and not used as direct comparison with competing methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httphhh..gps///ooo.......ppp/ooa
Answer: [Yes]
Justification: The paper conform in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Dataset is cited, URL is provided along with the license in Appendix A
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We share SAE checkpoints together with the concept dictionaries that we 've build. We also provide scripts load and use them at/attps:/githhb. bon/berktinaz//
ttable-concepts
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.

---

Appendix
A Additional details on SAE training
We obtain the learnable parameters of SAE by optimizing the reconstruction error
Lrea (W enc: Wecc. b) Cree (0) | 21
In practice, training only on the reconstruction error is insufficient due to the emergence of dead features. Dead features are defined as directions the latent space that are not activated for some specified number of training iterations resulting in wasted model capacity and compute. To resolve this issue, Gao et al. [12 proposes an auxiliary loss AuxK that models the reconstruction error of the SAE using the top-kouz feature directions that have been inactive for the longest. To be specific, define the reconstruction error as = x x; then the auxiliary loss takes the form
Lour (0) |le ell2
where is the approximation of the reconstruction error using the top-hoouz dead latents. The combined loss for the SAE training becomes
L (0) = Crec (0) + a.Ca (0),
where a is hyperparameter. We a filtered version of the LAION-COCO' dataset for training prompts. We train SAEs on the residual updates in the dif- fusion U-Net blocks down_blocks. attentions. mid_bli block. attentions.O, up_blue blockss. 1. attentions O. referred to as down_block, mid_block, up_block. We use guidance scale of w 7.5 and 50 DDIM steps to collect the activations. The dimension of the activation tensor is 16 16 x 1280 for down block and up block, 8 8 x 1280 for the mid_ block.. We train all models with Adam optimizer on single NVIDIA RTX A6000 GPU. Training on 200k prompts takes = 1 hour for up_ block and nown_block. and x 20 minutes for mid_bi block. Training hyperparameters are as follows:
a 32
setch_size:: 4096,
d 1280,
learning_rate, 0.0001,
kaur 256,
n_epochs:
n 4d = 51 20.
We keep track of normalized mean-squared error (MSE) and explained variance of the SAE recon- structions. In Iable[we we provide the complete set of training metries for combinations block, conditioning, timestep, and k.
B
Additional details on interventions
In table we provide the intervention strength (3) we use for each reverse diffusion stage and intervention type.
Table 2: Intervention strengths (8) for different intervention types and stages.
Intervention type
Stage
Intervention strength (9)
early 4000
spstially.targete
middle
400
final
1000
early
global
middle
10
final
10
license: apache-2.. 2.0
22

---

Table 1: Performance metrics for different block types, timesteps, conditioning and &s values, Best metrics for each conditioning block are underlined. Best overall metrics are bold.
Conditioning
Block
Timestep (t)
Scaled MSE
Explained Variance (%)
10
0.6293
36.6
down_loock
20
0,5466
44.8
10
0.6275
37.6
0.5
20
0,5510
45.1
cond
10
0,4617
1.0
0.3767 20
10
0.4817
50.5
mid_block
0,4133 20
57.3
0,4802 10
50.9
0.5
20
0,4194
57.0
10
0,4182
56.4
1.0
0.3503 20
63.3
10 0.5540
44.0
up_ block
0,4698 20
52.5
0.5414 10
45.3
0.5
0,4648 20
52.9
10
0.4177
57.7
1.0
0.3424 20
65.3
10 0.6306
36.4
block
0.5477 20
44.6
0.6364 10
36.9
0.5
20
0,5580
44.5
uncond
0,3874 10
58.6
1.0
0,3081 20
66.9
10 0.4852
50.7
mid_block block
0.4161 20
57.6
0.4909 10
50.8
0.5
20
0.4277
57.0
0,3286 10
65.7
1.0
0,2613 20
72.6
10
0,5550
44.0
up_block
20
0,4701
52.5
0,5436 10
0.5
20
0,4653
53.3
10
0,2724
71.4
1.0
20
0.2115
In fig. we perform global intervention concept #1222 (corresponding to cartoon look in top activating images) for all timesteps for various intervention strength 3.
C Additional results segmentation accuracy
We provide comprehensive overview of the accuracy of predicted segmentations across different architectural blocks SDv1.4 Figure
We find that coarse image composition can be extracted from any the investigated blocks, and from both cond and uncond features even in the first reverse diffusion step. We consistently observe saturation by the middle of the reverse diffusion trajectory. We note that the saturation is partially due imperfect ground truth masks from our annotation pipeline that can less accurate than the masks obtain from the SAE features at late time steps. Overall, up_ pl_ckock provides the most accurate, and mid_ block the least accurate segmentations (due to the lower spatial resolution the bottleneck). We observe consistently lower segmentation accuracy based on uncond features. We hypothesize that uncond features may encode more low-level visual information, whereas cond features are directly influenced by the text conditioning and therefore represent more high-level semantic information. Surprisingly, the reconstruction error however is higher for SAEs trained on cond features as depicted in Table
23

---

2- 0.0
3
3-20
- 4.01
- 10:0
8- 20.3
5 4001
(a) Etrly-stage intervention
(b) Middle-stage intervention
3
3
5 4001
(c) Final Final-stage intervention
Figure 7: Effect of intervention strength. We perform global intervention on a concept (#1722 for all time steps) corresponding to cartoon look top activating images. Early-stage interventions, at any strength, are unable to modify image style consistently but broadly influence image composition. Interventions in the middle stages imbue the image with the target style with increasing intensity. We only observe minor textural changes in final stages of diffusion, even at high intervention strengths.
24

---

0.31
0.35
0.32
191
loU
0.30
0.30
0.30
cood
0.28
cond
cond
0:25
uncond
0.20
uncond
0.25
uacond
1.0
0.5
0.0
10
0.5
0.0
1.0
0.5
0.0
Diffusion time
Diffusion timne
Diffusion timne
(a) down_block
(b) mid_block
(c) up_block
Figure 8: Accuracy of predicted segmentations based on SAE features from different architectural blocks. cond stands for text-conditioned diffusion features, and uncond denotes null-text condition- ing.
D Qualitative assessment of activations
We visualize the activation maps for top 10 (in terms of mean activation across the spatial dimensions) activating concepts for generated samples in Figures 9 [ various time steps and blocks. Based on our empirical observations, the activations can be grouped in the following categories:
Local semantics Most concepts fire semantically homogeneous regions, producing a semantic segmentation mask for particular concept. Examples include the segmentation the pavement, buildings and people in Figure 9 the plate, food items and background in Figure [0] and the face, hat, suit and background in Figure [1 We observe that these semantic concepts can redundant in the sense that multiple concepts often fire in the same region (e.g. Fig. I0 second row with multiple concepts focused the food in the bowl). We hypothesize that these duplicates may add different conceptual layers to the same region (e.g. food and round in the previous example). In terms of diffusion time, we observe that the segmentation masks are increasingly more accurate with respect to the final generated image, which is expected as the final scene progressively stabilizes during the diffusion process. This observation is more thoroughly verified in Section 4.2 and Figure [3a In terms of different U-Net blocks, the most accurate segmentation of the final scene, especially at earlier time steps.
observe that up blocks........ 1 attentions. 0 provides
Global semantics (style) We find concepts that activate more or less uniformly in the image. We hypothesize that these concepts capture global information about the image, such as artistic style, setting or ambiance. We observe such concepts across studied diffusion steps and architectural blocks.
Context-free We observe that some concepts fire exclusively in specific, structured regions of the image, such as particular corners bordering edges of the image, irrespective of semantics (see e.g. the last activation in the first row of Figure D We hypothesize that these concepts may be result of optimization artifacts, and are leveraged as semantic- independent knobs for the SAE to reduce reconstruction error. Specifically, if the SAE is unable "find" meaningful concepts in the image, as encouraged by the training objective. it may compensate for the missing signal energy in these context-free directions. Visual examples and further discussion can be found in Appendix G
E
Quantitative results
E.1
Quantifying temporal evolution of concepts
In order understand how much each SAE feature aligns with a particular concept beyond the qualitative results we have, we introduce two metrics: concept cohesion and concept separability. We refer to concept cohesion as measure of how tightly clustered items the concept dictionary are around their concept center. Similarly, we calculate concept separability as measure of how distinct concepts are from one another. Next, we talk about in detail how we calculate these scores. To calculate concept cohesion, for each concept, calculate the mean word2vec embedding of
25

---

down blocke attenticas
umet. .id. block. astentione
unet up blocks.
atsentionn
Figure 9: Visualization of top activating concepts in a generated sample. Concepts sorted by mean activation across spatial locations and top 10 activation maps are shown. Each row depicts a different snapshot along the reverse diffusion trajectory starting from pure noise (t 1.0) and terminating with the generated final image (t 0.0). Note that each row within the same column may belong to different concept, as concepts are not directly comparable across different diffusion time indices (separate SAE is trained for each individual timestep). Sample ID: 2000018.
the items in the concept dictionary. Then, for each item in the dictionary entry, we calculate the cosine distance to the mean embedding and take the average. The final score is calculated by taking the average across concepts. As for the concept separability, we first calculate the concept centers described before. Note that, have an X d tensor where n is the number of concepts in our dictionary and d is the word2vec embedding dimension. We then calculate the pairwise cosine similarity matrix (of shape n n). Then denote the final score the average of non-diagonal entries. We calculate these for our concept dictionaries for each timestep t E {0.0, 0.5, 1.0]. The results are presented in Table
Table 3: Concept cohesion and separability across timesteps f.
Concept Cohesion
Concept Separability
0.0
0.664 = 0.182
0.344 + 0.159
0.5
0.639 0.170
0.363 + 0.154
1.0
0.588 + 0.169
0.433 + 0.169
26

---

blocke. a.tenticcss....
umet. mid block. actentione...
unet up. blocks. diatsentione
Figure 10: Visualization of top activating concepts in a generated sample. Concepts sorted by mean activation across spatial locations and top 10 activation maps are shown. Fach row depicts different snapshot along the reverse diffusion trajectory starting from pure noise terminating with the generated final image (t = 0.0). Note that each row within the same column may belong different concept, concepts time indices (separate SAE is trained for each individual timestep). Sample ID: 2000035.
1.0) and
directly comparable across different diffusion
We observe that we move along the reverse diffusion iterates (1 1.0 to 0.0), the concepts that we have identified become purer (as measured by the increase concept cohesion from 0.588 to 0.664) Moreover, average pair-wise similarity of concepts decrease from 0.433 to 0.344. This indicates that concepts are becoming more distinct and separated as generation progresses.
E.2 Quantitative assessment of intervention success
Spatially targeted interventions To quantitatively assess the success spatially targeted inter- ventions summarized in Figure 4 we utilize CLIPSeg [29) as zero-shot segmentation method to determine whether the object in the generated image the intended quadrant. More specifically, using the object name, CLIPSeg gives prediction score for each pixel in the generated image. We calculate the center of the mass and determine which quadrant of the image belongs to. We run the experiments on an extended object list consisting of 10 in total ("bee", "book", "dog". "apple", "banana", "cat", "car". "phone", "door", "bird"). We select these objects as they are commonly observed in the LAION-COCO dataset. For each timestep, we calculate the overall spatial edit success as the average matching score across all objects and quadrants. We present the average score in T
27

---

down, blocke. 2. a.ttttnttcsss..
unet. nid block. actentione... .
unet up blocks. i.tteentions
1i. 0.5
Figure 11: Visualization of top activating concepts in a generated sample. Concepts sorted by mean activation across spatial locations and top 10 activation maps are shown. Each row depicts different snapshot along the reverse diffusion trajectory starting from pure noise terminating with the generated final image (t = 0.0). Note that each row within the same column may belong different concept, concepts not directly comparable across different diffusion time indices (separate SAE is trained for each individual timestep). Sample ID: 2000042.
1.0) and
Table 4: Performance of spatially targeted interventions across timesteps
Method
0.0
0.5
1.0
Random Baseline
0.25
0.25
0.25
Ours
0.23
0.35
0.80
We observe the edits at timesteps t 0.0 and t = 0.5 unsuccessful and to be comparable to the random prediction baseline. At 1.0, our edits are successful 80% of the time. It is likely that we can improve the performance further with hyperparameter tuning, which we leave it for future work.
Global interventions Since no ground truth images are available in the case of global interventions, we adopt a proxy metric to measure success (inspired by earlier focuued works such as Prompt- to-Prompt (5 CLIP similarity between
generated image and the mean CLIP embedding of the target concept (computed as the average embedding of all dictionary words associated with that concept). We measure this similarity both before and after the intervention.
edit is considered successful if the similarity increases post-intervention. We also report the average LPIPS score
28

---

between the image before and after intervention proxy to the quality of the generated image. That is, ideally we want the edited images to stay close the "reference" (before intervention) image.
To perform this analysis, we sample 1000 prompts from the validation set for added robustness and apply editing method three different diffusion timesteps (early, middle, late) for the concept indices highlighted in Figure 5 (which correspond to the concepts cartoon, sea & sand, and painting). The results are summarized in Table
Table 5: Global intervention performance across timesteps t.
Timestep
Concept ID
hefore hefore
CLIP before
ACLIP
Edit Success
LPIPS
t t= =.0.0
1722
0.201
0.208
+0,007
0.85
0.114
2349
0.190
0.194
+0.004
0.73
0.094
3593
0.201
0,206
+0.005
0.76
0.133
Average
0.197
0.203
+0.006
0.78
0.114
t= 0.5
1722
0.203
0.222
+0.019
0.94
0.379
524
0.192
0.220
+0,028
0.96
0.407
2137
0.203
0.218
+0.015
0.88
0.369
Average
0.199
0.220
+0.021
0.93
0.385
1 1.0
1722
0.202
0.212
+0.010
0.77
0.647
2366
0.194
0.197
+0.003
0.58
0.646
2929
0.203
0.212
+0,009
0.73
0.665
Average
0.200
0.207
+0.007
0.69
0.653
We find that edits most successful at the middle timestep (t 0.5), with 93% success rate and the highest average ACLIP similarity (+0.021), compared to 78% (+0.006) and 69% (+0.007) at early and late stages, respectively. This aligns well with our qualitative observations and supports our main insight: intervening during the middle stages of diffusion offers the most effective control over image style while preserving content. Furthermore, we observe mean LPIPS scores for / 0.0, t = 0.5, and t = 1.0 as 0.114. 0.385, and 0.653 respectively. Although LPIPS score of 0.114 is the best among diffusion stages, we note the low edit success. Note that this in line with our observation that "global interventions in the final stage of diffusion only results in minor textural changes without meaningfully changing image style". Similarly, we observe a large mean LPIPS score for early stage edits, likely due to major changes in the image composition. The middle stage LPIPS stands in between the two extremes but has significantly higher ACLIP and edit success.
Additional details on building concept dictionary and predicting image composition
F.1 Extracting interpretations from SAE features
In Figure 12
we provide a detailed figure to depict the curation of the concept dictionary (also explained Section 3.3) To build the concept dictionary, we first sample set of text prompts, generate the corresponding images using a diffusion model and extract the SAE activations for each CID during generation. We obtain ground truth annotations for each generated image using pre-trained vision pipeline, that combines image tagging, object detection and semantic segmentation, resulting in mask and label for each object in generated images. Finally, we evaluate the alignment berween our ground truth masks and the SAE activations for each CID, and assign the corresponding label to the CID only if there sufficient overlap.
F.2 Predicting image composition from SAE features
In Figure 13 we show the pipeline that we use predict image composition in detail. Leveraging the concept dictionary, we predict the final image composition based on SAE features at any time step, allowing us to gain invaluable insight into the evolution of image representations in diffusion
29

---

1. Generate image and cache SAE activations
2. Annotate image using zero-shot vision pipeline
flower
Tag Detect
phone
Segment
floor
3. Match activations with masks based on loU
4. Update concept dictionary with new labels
SAE
Annotated object masks
activations
CID
phone
Concept dictionary
CID 0: ground, dirt, floor
CID
flover
CID 1: phone case, screen, phone
CID
floor
CID 2: leaves, plant flower
Figure 12: Curating the concept dictionary: 1) We cache SAE activations across time steps and blocks during generation. 2) We leverage pipeline of image tagging, open-set object detection and promptable segmentation to annotate the generated image with segmentation masks and object labels.
3) We find SAE activations that overlap with the object masks. 4) We add the overlapping object's label to the concept dictionary under the matching SAE activation's 's CID.
models. Suppose that we would like to predict the location of a particular object in the final generated image, but before the reverse diffusion process is completed. First, given SAE features from a given intermediate time step, we extract the top activating concepts for each spatial location. Next, we create conceptual map of the image by assigning word embedding to each spatial location based on our curated concept dictionary. This conceptual map shows how image semantics, described by localized word embeddings, vary spatially across the image. Given a concept we would like to localize, such as object from the input prompt, we produce target word embedding and compare its similarity to each spatial location in conceptual map. To produce predicted segmentation map, we assign the target concept to spatial locations with high similarity, based on pre-defined threshold value. This technique can be applied to each object present the input prompt (or to any concepts of interest) to predict the composition of the final generated image.
F.3 More examples from the concept dictionary
We depict top 5 activating concepts, extracted from up_blocks. 1, a.ttentions O, for generated images and their corresponding concept dictionary entries in Figures 16
G Context-free activations
We observe the emergence feature directions in the representation space of the SAE that are local- ized to particular, structured regions in the image (corners, vertical or horizontal lines) independent of high-level image semanties. We visualize examples Figures 17 18 concept IDs for which the variance of activations averaged across spatial dimensions is minimal over validation split. We depict the mean and variance of such activations and showcase generated samples that activate the particular concept. We observe that these localized activation patterns
Specifically, we find
30

---

Extract top activating SAE features per location
H W
Get top concepts for each location
H
W
n
n
Prompt: dog the beach, chasing a ball
First diffusion step
2. Create conceptual map from SAE features
Conceptual map
CID ground, dirt
W
H
CID 68 sand, beach
Embed words
dw
Cuearpt derinnse
+ aggregate
CI Ergiraal
CID 97
mud, soil
CTD plea rar
CTD bem, yar
3. Predict final image segmentation from early concepts
Target word embeddings
Cosine similarities
Predicted segmentation
Final
image
dog
0.3
threshold
beach
0.8
+ argmax
ball
0.2
Figure 13: Predicting image composition: 1) We cache SAE activations at the first diffusion step (or other time step of interest) and extract top activated concepts per spatial location. 2) We fetch corre- sponding objects from the concept dictionary and produce conceptual embedding via Word2Vec. 3) We compare the conceptual embedding at each location the target word embeddings from the input prompt and predict segmentation map based on cosine similarity.
vellag, Iros, sadkpad, caaggy irig
CUR RII slordas eaake, ginn dod,
CTD: Iealowni
bakegg, talaee ielutrds, sakurtad ril
firel
balour; CID: bakiryy belsemuc, gartin
ameilat: with ratr deall
CTD: 1985
Gurss lone perk,
collag nakey coles, enarge nass wastoasn
lead, Sonno led. load pork.
1.0
(b) 0.0
Figure 14: Concept dictionary and visualization of the activation maps for the top 5 activating concepts. Sample ID: 2000031
31

---

CTD: 2218 test Sootell RMM hurgpet yoll gaon lana, ytzk, greo
CTDY 887
CID: 2010 Eutbed sxlk 940 your
lece Lazn looe
CITl: 487 whii Locae lnbking
Cit: 9TH Un
CEDs 2017
CID- /985
CIR STO? Joc. tekl kase, exilaa.
4228 #3E
(a) t 1.0
(b)
0.0
Figure 15: Concept dictionary and visualization the activation map for the top activating concepts.
Sample ID: 2000061
CLD: FZL Bal
CTTi 2004
valty
CDD: 1863
CD.- T0T imV
Viler the somee poid bater
CLD 2TD3 caSE
CID- 4180 perk, lord
CITi:
CID: 37521 c18
(a) 1.0
(b) 0.0
Figure 16: Concept dictionary and visualization of the activation map for the top 5 activating concepts. Sample ID: 2000062
32

---

Mean
Variance
#4181
10
#2200
20
-10
+ 156
-00
10
10
#877
20
#2220
10
#2247
20
#1
10
20
10
#662
#5000
#5555
20
Figure 17: We plot the mean and variance of activations, extracted at f = 1.0, for concepts with lowest average variance across spatial locations. We find concepts that fire exclusively at specific spatial locations. We depict generated samples that maximally activate for the given concept.
appear throughout the generative process (both at 1.0 in Figure 17 and at Moreover, the retrieved activating samples typically do not share common semantic or low-level visual features, as demonstrated by the sample images.
Figure 18).
H Visualization of top dataset examples
Top dataset examples for concept ID is determined by sorting images based their average concept intensity "e where the averaging is over spatial dimensions. Formally (definition is taken from (4 for transformer block & and timestep we define To as:
Σ j..li...c..
H.We 1.
In Figures t's.
19
32
we provide top activated images for various concept and for various timestep
33

---

Mean
Variance
#2558
#4482
#-6551
-20
-0
F-20
#1149
10
-2
#229
100
20
#122449
#44878
-10
#2775
1000
#1068
#2729
100
Figure 18: We plot the mean and variance of activations, extracted at f = 0.0. for concepts with lowest average variance across spatial locations. We find concepts that fire exclusively at specific spatial locations. We depict generated samples that maximally activate for the given concept.
34

---

SIVER GLNCEE GOWDO!
COMARE IN SUMPEN TE TIVE COMES
LOVE SEVES.
THE SO DOE STAR STR
SLARMUY DE DONG
HO!
Figure 19: Top activating dataset examples for the concept ID 10 belonging to the SAE trained on the cond activation of mid_block at 1.0.
Exprister TO ANEEAD JA BAOD BOZE ALET LE
AP
f Al x
サ
SATA
ANTAN SATE
STALTI
MONIBET OHR BOOH BOME SIRRE
The Boooke Revitvierew
Bnstorioe boudiot. the auminiaa porn verr
Figure 20: Top activating dataset examples for the concept ID 13 belonging to the SAE trained on the cond activation of mid_block block at t 1.0.
35

---

dE LIME'S S AMEE
THERT PIAPRST'S PORE CTS,
Eedlis
"JOETNP
Kingdaoom Bisds ddra
Drig DAGINON/TOO
Drut
Figure 21: Top activating dataset examples for the concept ID 49 belonging to the SAE trained on the cond activation of mid_ block at 1.0.
36

---

NENE
Figure 22: Top activating dataset examples for the concept ID 1722 belonging to the SAE trained on the cond activation of at t = 1.0.
37

---

Io cau
The Danty WO Eor--ro
SETEAM
Figure 23: Top activating dataset examples for the concept 2787 belonging to the SAE trained on the cond activation of mid_block at t = 1.0.
38

---

PEARY OE ALCAS
Figure 24: Top activating dataset examples for the concept ID 524 belonging the SAE trained on the cond activation of mid_block at t = 0.5.
39

---

V2EE olf Torchaod
Figure 25: Top activating dataset examples for the concept ID 1314 belonging to the SAE trained on the cond activation of mid__bock at t = 0.5.
40

---

60
XOXX
Figure 26: Top activating dataset examples for the concept ID 1722 belonging to the SAE trained on the cond activation of mid_block at t = 0.5.
41

---

Figure 27: Top activating dataset examples for the concept 2137 belonging to the SAE trained on the cond activation of mii_block at t = 0.5.
Figure 28: Top activating dataset examples for the concept ID belonging to the SAE trained on the at t 0.0.
cond activation of
I Broader Impact Statement
In this work, we demystify the inner workings of text-to-image diffusion models by revealing how visual representations evolve over the reverse diffusion process. By uncovering interpretable features and developing stage-specific editing techniques, our approach may enable more granular and controllable image generation, which can benefit safety critical applications. However, we acknowledge that greater interpretability and control also carry risks, such as facilitating misuse for deceptive content generation or circumventing safety mechanisms.
42

---

Figure 29: Top activating dataset examples for the concept 1722 belonging to the SAE trained on the cond activation of mid_ block at / = 0.0.
LO Budion
WETER GTAZE SIVE FIRE
prihis Culi Vitichury
Vear for twns
VRE TTREN WYAND
otianter
Honiey Cumer
BUETIIE
Sostijiecnl Wwibhaddeon
ario falk Más
Wandword
ANUE INGATIA
Cy fatw
he Vax Sitet
A&S
Figure 30: Top activating dataset examples for the concept 4972 belonging to the SAE trained on the cond activation of mid_ block at t = 0.0.
43

---

Figure 31: Top activating dataset examples for the concept 4979 belonging to the SAE trained on the cond activation of mid_ block at / = 0.0.
Figure 32: Top activating dataset examples for the concept ID 86 belonging the SAE trained on the cond activation of down_blooo block at t = 0.0.


Reviews:
[{"review_id": "jsKUNo2xs4", "reviewer": "Reviewer_HyPM", "summary": "This paper investigates the internal representations learned by large text-to-image diffusion models, aiming to uncover the formation of human-interpretable concepts and to illuminate how prompts are gradually transformed into coherent images. To this end, the authors utilize sparse autoencoders (SAEs) --- which have recently emerged as powerful tools for mechanistic interpretability (MI) to discover highly interpretable features within large models at scale. Specifically, they train SAEs on residual stream activations within the U-Net backbone of the denoising model. By training these SAEs at multiple diffusion timesteps and transformer layers, the authors construct a collection of latent directions, each approximately aligned with a human-interpretable concept, that can be used to analyze, predict, and even edit the model’s outputs in a time-aware fashion.\n\nThe core of the paper is a method for constructing a temporally and spatially resolved “concept dictionary” by training SAEs on residual stream activations at selected layers and timesteps of the diffusion model. The autoencoders are designed to produce a fixed number of active latent features per input (via Top-K sparsity), with each latent dimension corresponding, as directly as possible, to a distinct human-interpretable concept. To annotate these features without relying on language models, the authors introduce a fully vision-based labeling pipeline that combines image tagging, open-set object detection, and segmentation to generate dense, spatially grounded concept masks. These labeled concepts are then embedded into a common vector space to enable layout prediction and concept matching. By applying this framework across different diffusion timesteps, the authors trace the emergence of semantic structure in the model’s internal representations and demonstrate how targeted interventions---adding or removing specific concepts at specific times---can causally manipulate the resulting image. Compared to prior approaches, this method offers finer spatial resolution, avoids language-induced bias, and enables time-aware interpretability and control within diffusion models.", "questions": "1. The paper assumes that each SAE feature aligns with a distinct, human-interpretable concept, yet some features appear polysemantic or noisy. Can the authors provide more quantitative evidence regarding the purity or disentanglement of the learned features? For example, have they measured how often individual SAE units correspond to a single label versus multiple conflicting ones?\n\n2. To what extent do the findings generalize beyond Stable Diffusion v1.4? For instance, would similar temporal patterns of concept emergence (layout $\\to$ style $\\to$ texture) appear in other architectures or in models trained on different datasets or prompts? Any experiments or ablations supporting generality would help clarify the broader significance of the results.\n\n3. The paper raises four insightful questions in the Introduction but provides relatively diffuse answers. Could the authors summarize, more explicitly, what new understanding these experiments yield? What are the actionable or conceptual takeaways (beyond mechanistic summaries of the observations), and how might they inform future model design or interpretability tools?\n\n4. The causal interventions via SAE directions are visually compelling but largely shown through examples. Can the authors provide quantitative metrics (e.g., success rates, semantic preservation, edit consistency) to assess how robust or reliable these interventions are across prompts and diffusion trajectories? How sensitive are the edits to the choice of timestep, layer, or scaling parameter?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- The paper addresses an important and increasingly relevant problem: understanding the internal workings of diffusion models, which have quickly become a dominant paradigm in generative modeling. The authors clearly motivate the need for time-resolved interpretability and go beyond superficial inspection by constructing actionable tools for probing and editing model behavior.\n\n- The use of sparse autoencoders is well-justified, with architectural and optimization choices grounded in prior interpretability work. The decision to focus on the residual stream (as opposed to raw hidden states) is conceptually sound, as this stream captures the new information injected at each step.\n\n- While there has been a surge in interpretability research, this paper stands out by emphasizing the temporal evolution of representations in diffusion models. The finding that layout solidifies early, style emerges midway, and texture is refined in the final steps is intuitive but now quantitatively substantiated.\n\n### Weaknesses:\n- **Assumption of semantic alignment.** The paper assumes that each SAE feature captures a single interpretable concept, but in practice some features appear polysemantic or distributed. While the authors attempt to filter and tag such features, the actual purity of the learned directions remains somewhat unclear---especially at higher sparsity levels or in semantically ambiguous cases. Furthermore, although the authors briefly mention the “superposition hypothesis” as a core assumption behind SAEs, a more explicit explanation of what this entails and why it is reasonable in this setting would be helpful.\n\n- **Questionable universality and takeaways.** While this work advances our understanding of text-to-image diffusion models through the lens of mechanistic interpretability---particularly in tracing how visual concepts evolve over the generative process---it remains unclear whether the observed patterns generalize to other architectures, datasets, or configurations. Moreover, although the paper gestures to answer the four motivating key questions posed in the Introduction, these answers remain somewhat implicit or tentative. It would be helpful to see clearer articulation of the takeaways and their broader implications for design or analysis of generative models.\n\n- **Evaluation scope and soundness.** The experimental results compellingly illustrate the emergence of semantic structure and the feasibility of targeted interventions. However, the evaluation remains somewhat qualitative and anecdotal. Quantitative metrics (e.g., edit success rates, feature purity scores, generalization to unseen prompts or styles) are limited, making it difficult to assess the robustness, reliability, or practical utility of the proposed methods. A more rigorous and systematic evaluation would strengthen the empirical foundation of the paper’s claims.", "was_revised": true, "final_justification": "The authors addressed most of my concerns and questions, and thus, I maintain my modestly positive evaluation.", "rebuttal": "We appreciate the time and effort the reviewer put into reviewing our manuscript. We are glad to hear that the reviewer thinks our work addresses “an important and increasingly relevant problem”  and that the paper “stands out by emphasizing the temporal evolution of representations”. Please find below our responses to the concerns and questions raised by the reviewer.\n\n- **Re question 1 and weakness 1:** We thank the reviewer for raising this important question. We agree with the reviewer that a quantitative and temporal analysis of concept purity would be beneficial to understand how much each SAE feature aligns with a particular concept beyond the qualitative results we have. To this end, we introduce two metrics: concept cohesion and concept separability. We refer to concept cohesion as a measure of how tightly clustered the items in the concept dictionary are around their concept center. Similarly, we calculate concept separability as a measure of how distinct concepts are from one another. Next, we talk about in detail how we calculate these scores.\nTo calculate concept cohesion,  for each concept, we calculate the mean word2vec embedding of the items in the concept dictionary. Then, for each item in the dictionary entry, we calculate the cosine distance to the mean embedding and take the average. The final score is calculated by taking the average across concepts. As for the concept separability, we first calculate the concept centers as described before. Note that, we have an $n \\times d$ tensor where $n$ is the number of concepts in our dictionary and $d$ is the word2vec embedding dimension. We then calculate the pairwise cosine similarity matrix (of shape $n \\times n$). Then we denote the final score as the average of non-diagonal entries. We calculate these for our concept dictionaries for each timestep (t=0.0, 0.5, 1.0). The results are presented in the following tables:\n\n  | t   | concept cohesion | concept separability |\n  | --- | ---------------- | -------------------- |\n  | 0   | 0.664            | 0.344                |\n  | 0.5 | 0.639            | 0.363                |\n  | 1.0 | 0.588            | 0.433                |\n\n  We observe that as we move along the reverse diffusion iterates (t=1.0 to t=0.0), the concepts that we have identified become purer (as measured by the increase in concept cohesion from 0.588 to 0.664) Moreover, average pair-wise similarity of concepts decrease from 0.433 to 0.344. This indicates that concepts are becoming more distinct and separated as generation progresses, in other words more specific. We thank the reviewer for the suggestion. We believe that these numbers complement our analysis well and we will incorporate these results in the revised manuscript. With respect to the superposition hypothesis, the high explained variance we observe during training combined with the enforced sparsity underpins the existence of an overcomplete basis where model activations can be decomposed into a few high-dimensional feature vectors. Our framework then labels these feature vectors (or concept vectors) with human-interpretable concepts.\n\n- **Re question 2 and weakness 2:** Great point. We agree with the reviewer that analysis on more models would be beneficial. Collecting activations for millions of text prompts is costly time and storage wise. We did our best to perform a comprehensive set of experiments on the Stable Diffusion v1.4 architecture and provide the first SAE based temporal analysis of the evolution of concepts. We would like to explore other architectures such as FLUX in future work.\n\n- **Re question 3 and weakness 2:** Let us clarify the main takeaways from our experiments based on the temporal stages:\n\n  - Early stage of diffusion: coarse image composition emerges as early as during the very first diffusion step. At this stage, we are able to approximately identify where prominent objects will be placed in the final generated image. Moreover, image composition is still subject to change: we can manipulate the generated scene by spatially targeted interventions that amplify the desired concept in some regions and dampens it in others. However, we are unable to steer image style at this stage using our global intervention technique. Instead of high-level stylistic edits, these interventions result in major changes in image composition.\n  - Middle stage of diffusion: image composition has been finalized at this stage and we are able to predict the location of various objects in the final generated image with high accuracy. Moreover, our spatially targeted intervention technique fails to meaningfully change image composition at this stage. On the other hand, through global interventions we can effectively control image style while preserving image composition, in stark contrast to the early stages.\n  - Final stage of diffusion: Image composition can be predicted from internal representations to very high accuracy (empirically, often higher than our pre-trained segmentation pipeline), however manipulating image composition through our spatially localized interventions fails. Our global intervention technique only results in minor textural changes without meaningfully changing image style. \n\n  The practical implications of this work are far-reaching. First, it informs practitioners when to intervene effectively to manipulate the behavior of diffusion models, not only for image editing but also for generated content moderation. For instance, to avoid copyright infringements, interventions on filtering protected artistic style is most effective in the middle timesteps. Second, our framework opens the door for extremely fine-grained image manipulation through natural language instructions. In particular, the concept dictionary is a library of tuning knobs expressed in natural language. As future direction, an LLM can act as a controller that maps editing instructions from natural language to interventions based on the concept dictionary (similarly how a human would browse the concept dictionary to implement their edit).  \n- **Re question 4:** We conducted a new experiment to quantitatively evaluate the success of our editing method. Since no ground truth images are available, we adopt a proxy metric: CLIP similarity between the generated image and the mean CLIP embedding of the target concept (computed as the average embedding of all dictionary words associated with that concept). We measure this similarity both before and after the intervention. An edit is considered successful if the similarity increases post-intervention.\nTo perform this analysis, we sample 100 prompts from the validation set and apply our editing method at three different diffusion timesteps (early, middle, late) for the concept indices highlighted in Figures 4, 5, and 6 (these correspond to the concepts cartoon, sea & sand, and painting, respectively). The results are summarized in the following table:\n\n  | Timestep | Concept ID | Before Edit | After Edit | Δ (After - Before) | Edit Success |\n  |----------|------------|-------------|------------|---------------------|---------------|\n  | t = 0.0  | 1722       | 0.198       | 0.206      | +0.008              | 0.82          |\n  |          | 2349       | 0.190       | 0.193      | +0.003              | 0.69          |\n  |          | 3593       | 0.198       | 0.204      | +0.006              | 0.79          |\n  | **Average** |            | **0.195**   | **0.201**  | **+0.006**          | **0.77**      |\n  | t = 0.5  | 1722       | 0.198       | 0.222      | +0.024              | 0.97          |\n  |          | 524        | 0.190       | 0.220      | +0.030              | 0.95          |\n  |          | 2137       | 0.198       | 0.219      | +0.021              | 0.93          |\n  | **Average** |            | **0.195**   | **0.220**  | **+0.025**          | **0.95**      |\n  | t = 1.0  | 1722       | 0.198       | 0.211      | +0.013              | 0.84          |\n  |          | 2366       | 0.190       | 0.193      | +0.003              | 0.56          |\n  |          | 2929       | 0.198       | 0.211      | +0.013              | 0.78          |\n  | **Average** |            | **0.195**   | **0.205**  | **+0.010**          | **0.73**      |\n\n  We observe that stylistic edits are most successful when applied in the middle timestep (t = 0.5), achieving a 95% success rate, compared to 77% and 73% at early and late stages, respectively. This is also reflected in the highest mean ΔCLIP similarity (+0.025), compared to +0.006 and +0.010 at the other timesteps. These findings align with our qualitative observations and reinforce our key insight regarding the middle stage of diffusion: global interventions at this point enable effective control over image style. Combined with our additional experiments on concept cohesion and separability, we believe this quantitative evidence strengthens our analysis which we will incorporate into the revised manuscript.\n\nWe hope our responses have adequately addressed the reviewer’s concerns. We welcome further discussion during the rebuttal period should any issues remain."}, {"review_id": "o1hYK3A5aZ", "reviewer": "Reviewer_Pskz", "summary": "This work provides a mechanistic interpretability framework based on sparse autoencoders to discover and study concept evolution across time steps in diffusion models. Spatial and global inntervenctions are performed to perform image manipulation based on local and style concepts. The dataset considered for the study is LAION-COCO.", "questions": "Questions:\n1. How does the concept dictionary generalize and cater to the rare concepts?  What is the granularity of the concepts that can be discovered by the proposed framework?\n2. How does the novelty of the approach standout given the research questions on  level of image representation is present in the early stage of the generative process, evolution of visual representations and the time-evolution have been previously studied and applied to various vision tasks such as segmentation. The methods should also be cited and differences in the findings should be discussed\n3.  How is the ideal tilmestep chosen for intervention in figures 4,5, and 6?\n4. What defines the image style in the concept dictionary?  Image style if specific to an object (textures) can be low-level.\n5. Quantifying the goodness of concepts. How can once validate that the concepts recovered are actually meaningful for the given image if no domain knowledge is provided eg, difference between anime style or cartoon style.", "limitations": "Limitations have been addressed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The work provides a framework based on SAEs to analyze the evolution of visual representations in text-to-image diffusion models. \n2. The work also considers the intervention techniques to manipulate the style or layout of the images. The intervention strength is made adaptive to account for the activations of different features.\n3. The paper is easy to follow and provides examples to validate their approach.\n\nWeaknesses:\n1. The findings of the paper though interesting, are not new and have been previously researched. For example, the evolution of visual features with time steps for text-conditioned diffusion models has been analyzed in (Mahajan etal and Patashnik etal) showing how high-level and low-level semantics evolve. Prompt+ shows how the different blocks of U-Net are sensitive to different image features. The existence of image information at t=0 is also not new and has been \n2. The dataset used in the study is LAION-COCO which is also the dataset that models like SD1.4 (used in the paper) train on. How does this interpretability framework generalize concept discovery?\n3. The framework relies on manual inspection to select time steps to be used for global/local manipulation. \n4. In figure 4, the suggested interventions do not preserve the identity of the object's book and the dog; this shows that semantics do change for some classes in the early stages. \n5. It is not clear what property makes the intervention in section 3.5 causal. The normalization to the intervention strength are applied to global intervention or to both spatial and global; the equations should be updated accordingly.\n\nPatashnik et al: Localizing Object-level Shape Variations with Text-to-Image Diffusion Models.\nMahajan et al: Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models.", "was_revised": true, "final_justification": "The work makes solid contribution to understanding of diffusion models leveraging a framework based on SAEs to analyze the evolution of visual representations in text-to-image diffusion models. The approach offers a streamlined procedure to understand the concepts across time-steps.", "rebuttal": "Thank you for taking the time to review our manuscript and provide valuable feedback. We are glad to hear that the reviewer appreciates our analysis on interventions, found it easy to follow the paper and that the reviewer believes our examples validate the proposed approach. Please find below our responses to the concerns.\n\n- **Re question 1:** With respect to novel concepts, by construction, our framework can discover concepts based on natural language annotations of image regions. Thus, given such annotations for a novel concept, we can incorporate it into the concept dictionary in the first step of our method by observing which features fire for the given annotated samples. With respect to the granularity of discovered concepts, we refer the reviewer to figures 14,15,16 in the Appendix where we provide snippets from our concept dictionaries. Based on the recommendation of Reviewer HyPM, we run experiments to quantitatively measure the purity and separation of the concepts that we discover and how it temporally evolves across diffusion iterates. We observe that the objects in the dictionary cluster around the concept center better (increased concept cohesion) and that the concepts are separated better (more distinct) as we progress along the reverse diffusion iterates. We refer the reader to our response to “question 1 and weakness 1” of Reviewer HyPM for the full experimental details and results table.\n- **Re question 2 and weakness 1:** We are unaware of prior work that highlights the presence of high-level concepts in the early stages of diffusion with the granularity that our technique provides, namely decoding the final generated image layout before the first diffusion step. Even though there is existing work that shows that image layout can be manipulated in early timesteps, our findings are stronger: we can localize and label the concepts, which is a key difference. Moreover, our SAE-based technique provides a more structured analysis on the emergence of visual features and their granularity that provides further explanation and corroboration of phenomena observed empirically in existing work. \n- **Re question 3 and weakness 3:** We would like to clarify that we do not manually inspect the diffusion timesteps to select an intervention time. A priori, we split the diffusion iterates into 3 stages: early (t ∈ [0.6, 1.0]), middle (t ∈ [0.2, 0.6]) and final (t ∈ [0, 0.2]). We use these fixed intervals for all of our experiments and in Figures 4,5,6.\n- **Re questions 4:** We define the image style of a concept in the dictionary as the shared visual characteristic among its top-activating examples from the dataset. Indeed, if the shared characteristics are low-level (such as texture), we define the style accordingly. \n- **Re question 5:** One can still examine the top-activating examples from the dataset for a given concept to get a sense of its visual characteristics without needing to assign a specific label (e.g., \"anime style\" or \"cartoon style\"). In this way, domain knowledge is not required to visually compare a candidate image to the set of images associated with a concept. That said, domain knowledge may be necessary to accurately name or describe the style represented by that set.\n- **Re weakness 2:** Great question. We use the set of prompts from the LAION-COCO dataset to generate images, which are then annotated by our zero-shot pipeline, thus we are not leveraging LAION-COCO images directly at all when building the concept dictionary. As we have explained earlier, given samples of a novel concept and annotated image regions, we can incorporate such concepts into the concept dictionary.\n- **Re weakness 4:** Great point. We hypothesize that early stage concepts do not have enough granularity to encode different dog breeds as separate concepts. Instead, a unified “dog-ness” is encoded in the concept dictionary. Therefore, manipulations to such a concept do indeed change the location of the “dog” to the target spatial location but cannot give guarantees to preserve high granularity details.\n- **Re weakness 5:** We refer to our interventions in Section 3.5 as causal because our goal is to verify that the discovered concepts have a visual impact when we manipulate them through targeted interventions. We apply slightly different normalizations for spatially targeted and global interventions. We will clarify this better in the revised manuscript.\n\nPlease let us know if you have any unanswered questions or concerns with respect to our work. We are happy to discuss further in the discussion period."}, {"review_id": "PICBHjGpSq", "reviewer": "Reviewer_7YW2", "summary": "This paper proposes to analyze the diffusion process by training sparse autoencoders on the model activations from different timesteps.  Specifically, this paper finds that image composition emerges at the early steps of the diffusion step. Based on the analysis, this paper proposes an intervention method that manipulates image composition and style in image generation.", "questions": "Please refer to the weaknesses.", "limitations": "This paper discusses the limitations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper proposes to use sparse autoencoders (SAE) to analyze the generative process of diffusion models, which is interesting. \n2. This paper proposes a new method to manipulate the image composition and style using SAE features.\n3. Experiments show the effectiveness of the proposed method when interventing at the early and middle stages of the diffusion process.\n\n**Weaknesses:**\n\n1.  The findings about what diffusion models learn at different timesteps are kind of unsurprising given the prior work such as [1]. For example, prior work [1] has shown that diffusion models tend to capture coarse features (e.g., global color and structure) in early steps, then learn the content, and refine details in later steps.  This limits the contribution of the claims. \n\n[1] Perception Prioritized Training of Diffusion Models, CVPR 2022.\n\n2. A much simpler method to analyze the image composition is to visualize the generated images across different timesteps directly. For example,  Figure 2 in [1] shows that diffusion models have learned the spatial composition and structure at the early stages of diffusion process, which makes discussions in Section 4 less surprising.\n\n3. This paper presents the experimental results only through qualitative visualizations, such as Figure 4,5.6. How about the quantitative metrics evaluated on the large dataset? Moreover, how does the proposed intervention method perform when comparing to prior image editing methods? These results would strengthen and better clarify the contribution of this paper.", "was_revised": true, "final_justification": "I have carefully written my reviews and discussed with the authors. I have also read the comments from other reviewers. I agree that this work proposes an interesting method to understand the learned concepts in diffusion models via SAE.  The authors also add some quantitative results during the rebuttal, which solved some of my earlier concerns. \n\nHowever, one key concern remains unresolved. As stated in the first weakness point of my initial review, the temporal analysis findings in this work are similar to prior studies (e.g., [1] I provided in the initial review and other related works [2,3,4] I provided in my follow-up response). Specifically,  this work finds that image composition can be effectively controlled in the early diffusion stages and image style in the middle stages. This finding is not surprising given the findings of prior studies[1,2,3,4]. The authors discussed this part in the first-round rebuttal, and I pointed out my further concern and provided more related work[2,3,4] in my response. Unfortunately, the authors’ replies did not adequately address this issue.\n\nGiven the lack of a significant difference between the findings of this work and those of prior studies,  more critical and clearly different findings would be necessary to meet the acceptance standards of NeurIPS. Therefore, I will maintain my original score.\n\nI quote my comments when discussing with the authors regarding this issue as follows:\n\n> I appreciate the authors' effort in discussing the difference from [1] in the rebuttal, and I noted their claim that the work highlights \"decoding the final generated image layout before the first diffusion step\". However, I remain unconvinced that this is a critical finding for highlighting the first diffusion step, instead of early denosing steps in prior studies[1,2,3]. In fact, this work itself performs interventions across a range of early steps (see line 275), which further weakens the argument that the first step is uniquely important.\n\n> The temporal analysis of diffusion models has already been widely discussed prior studies, including but not limited to the works listed below. For instance, Prompt-to-Prompt (with over 2,000 citations) shows in Figure 4 that object layouts is decided in early timesteps. SDEdit [3]  discusses how selecting different starting timesteps $t$ influences image layout and style. Another work [4] investigates how the editing strength varies with different diffusion timesteps. \n\n[1] Perception Prioritized Training of Diffusion Models, CVPR 2022\n\n[2] Prompt-to-Prompt: Image Editing with Cross Attention Control\n\n[3] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR 2022\n\n[4] Diffusion Models Already Have a Semantic Latent Space, ICLR 2023", "rebuttal": "We appreciate the reviewer’s time and thoughtful feedback on our manuscript. We're glad the reviewer found the use of SAEs to analyze the diffusion process “interesting” and that the experiments demonstrate the effectiveness of our method.\n\n- **Re weakness 1:** As we have explained in our response to Reviewer Pskz, we are unaware of prior work that highlights the presence of high-level concepts in the early stages of diffusion with the granularity that our technique provides, namely decoding the final generated image layout before the first diffusion step. Even though there is existing work that shows that image layout can be manipulated in early timesteps, our findings are stronger: we can localize and label the concepts, which is a key difference. Moreover, our SAE-based technique provides a more structured analysis on the emergence of visual features and their granularity that provides further explanation and corroboration of phenomena observed empirically in existing work. \n- **Re weakness 2:** We appreciate the reviewer’s suggestion and agree that visualizing generated images across timesteps can offer insight into the evolution of image structure during the diffusion process. However, we would like to clarify an important distinction: our work uses a latent diffusion model (LDM), in which the diffusion process occurs in a latent space rather than pixel space. As shown in Figure 1 of our paper, the latent representations at early timesteps (e.g., t = 0.0) resemble noise and do not convey any obvious spatial composition when visualized via the decoder's posterior mean. This makes it much harder—if not impossible—to directly infer spatial structure at these stages using standard visualization. Even if it is possible to obtain low-quality representation of the diffusion model’s output at different time steps, it is unclear how to leverage them for fine-grained concept discovery, as out-of-the-box annotation tools typically only work on clean images. \n- **Re weakness 3:** We thank the reviewer for highlighting the importance of including quantitative results to support our qualitative findings. In response (echoing suggestions from other reviewers as well) we conducted two sets of new experiments to strengthen our analysis.\n\n  *Quantitative Characterization of Concept Dictionaries*\n\n  To quantify the structure of concept representations discovered at different timesteps, we define two metrics: concept cohesion and concept separability.\n  - Concept cohesion measures how tightly clustered the items in a concept dictionary are around their respective concept centers.\n  - Concept separability quantifies how distinct the concepts are from each other.\n\n  To compute cohesion, we take the mean Word2Vec embedding of the dictionary items for each concept, then calculate the average cosine distance between individual items and the concept center. The final cohesion score is the average of these values across all concepts.\n\n  For separability, we compute the pairwise cosine similarity matrix of the concept centers (shape $n \\times n$  where $n$ is the number of concepts). We then report the average of the off-diagonal similarities, representing the average similarity between different concepts. The results for each timestep are summarized below:\n\n  | t   | concept cohesion | concept separability |\n  | --- | ---------------- | -------------------- |\n  | 0   | 0.664            | 0.344                |\n  | 0.5 | 0.639            | 0.363                |\n  | 1.0 | 0.588            | 0.433                |\n\n  We observe that, as we move along the reverse diffusion process (from $t=1.0$ to $t=0.0$), concepts become more coherent and less similar to each other—indicating that they become purer and more distinguishable (that is more specific) as generation progresses toward the clean data distribution. These results support our interpretation of the evolving semantic structure of concepts over time.\n\n  *Quantitative Evaluation of Editing Performance*\n\n  To evaluate the effectiveness of our intervention method, we introduce a proxy metric based on CLIP similarity. Specifically, we compute the CLIP distance between a generated image and the mean CLIP embedding of the target concept (obtained by averaging embeddings of all associated dictionary terms). An edit is considered successful if this similarity increases after intervention.\n\n  We conducted this experiment on 100 prompts from the validation set and applied our editing method at three timesteps (early, middle, late), focusing on concept indices corresponding to cartoon, sea & sand, and painting (as shown in Figures 4, 5, and 6). Results are as follows:\n\n  | Timestep | Concept ID | Before Edit | After Edit | Δ (After - Before) | Edit Success |\n  |----------|------------|-------------|------------|---------------------|---------------|\n  | t = 0.0  | 1722       | 0.198       | 0.206      | +0.008              | 0.82          |\n  |          | 2349       | 0.190       | 0.193      | +0.003              | 0.69          |\n  |          | 3593       | 0.198       | 0.204      | +0.006              | 0.79          |\n  | **Average** |            | **0.195**   | **0.201**  | **+0.006**          | **0.77**      |\n  | t = 0.5  | 1722       | 0.198       | 0.222      | +0.024              | 0.97          |\n  |          | 524        | 0.190       | 0.220      | +0.030              | 0.95          |\n  |          | 2137       | 0.198       | 0.219      | +0.021              | 0.93          |\n  | **Average** |            | **0.195**   | **0.220**  | **+0.025**          | **0.95**      |\n  | t = 1.0  | 1722       | 0.198       | 0.211      | +0.013              | 0.84          |\n  |          | 2366       | 0.190       | 0.193      | +0.003              | 0.56          |\n  |          | 2929       | 0.198       | 0.211      | +0.013              | 0.78          |\n  | **Average** |            | **0.195**   | **0.205**  | **+0.010**          | **0.73**      |\n\n  We find that edits are most successful at the middle timestep (t = 0.5), with a 95% success rate and the highest average ΔCLIP similarity (+0.025), compared to 77% (+0.006) and 73% (+0.010) at early and late stages, respectively. This aligns well with our qualitative observations and supports our main insight: intervening during the middle stages of diffusion offers the most effective control over image style while preserving content. Moreover, we would like to emphasize that our framework is not specifically designed for image editing. Our work is aimed at uncovering the evolution of interpretable concepts in the representation space of the diffusion model, incorporating analysis of interventions to demonstrate the causal effect of manipulating concepts on the model’s output. That said, our framework opens the door for extremely fine-grained image manipulation through natural language instructions. In particular, the concept dictionary can be viewed as a library of image editing knobs expressed in natural language. As an exciting future direction, an LLM can act as a controller that maps editing instructions from natural language to interventions based on the concept dictionary (similarly how a human would browse the concept dictionary to implement their edit).\n\n  We thank the reviewer for prompting these additions and will incorporate both sets of results into the revised manuscript to clarify and reinforce our contributions.\n\n\nIf there are any aspects of our response that remain unclear, we welcome further discussion and would be glad to address them."}, {"review_id": "umO51eUbVx", "reviewer": "Reviewer_ZsBz", "summary": "This paper uses Sparse Autoencoders (SAEs) to analyze the representation of Stable Diffusion v1.4, with the goal of understanding how interpretable concepts emerge and evolve throughout the multi-step diffusion process. The authors train SAEs on residual updates from cross-attention blocks at different timesteps ([0.0, 0.5, 1.0]) and develop a pipeline to interpret learned features. The authors argue that image composition emerges as early as the first diffusion step, and that different diffusion stages are responsible for different aspects of generation (composition vs. style).", "questions": "I will emphasize 3 areas of improvement:\n\n- Better comparison with prior work, and improved motivation of the automated interpretability pipeline. As stated in the weaknesses section, multiple claims regarding this pipeline are not motivated by any experiments.\n- Limited scope and generalization: many new diffusion models have been released since Stable Diffusion 1.4. To make such a strong claim (title) of analyzing the \"evolution of concepts in diffusion models\", you should demonstrate your findings are robust across multiple diffusion models. Additionally, motivating the choice of hyperparameters is important.\n- Motivation of the temporal analysis. It is currently not clear what is gained by a temporal analysis compared to a single, or few-steps analysis, such as performed in prior work.", "limitations": "No. I have listed many limitations that have not been discussed by the authors in their manuscript.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "## Strengths\n**Surprising Early Emergence Finding**\nThe discovery that \"even before the first reverse diffusion step is completed, the final composition of the scene can be predicted surprisingly well\" (Abstract, lines 12-15) is surprising. Figure 1 shows that semantic layout can be predicted from t=1.0 activations.\n\n\n## Weaknesses\n\n**Background and style**\n- The background sections on SAE and diffusion are too succinct. While it is not expected of the authors to write a full tutorial, the current version is not sufficient, especially for readers that are new to the field. This forces the authors to define concepts such as classifier-free guidance, specific blocks of the neural network, and the concepts of an encoder and decoder of the SAE in the methods section, which is not appropriate.\n\n\n**Lack of Justification for Temporal Analysis**\n- Lines 107/108: \n> First, we analyze the time-evolution of interpretable concepts during generation – critical for understanding and controlling diffusion – which single-step models cannot capture.\n- The paper claims that analyzing the time-evolution of interpretable concepts is \"critical for understanding and controlling diffusion,\" but does not provide a clear justification for this statement. What specific advantages or new insights does this temporal analysis offer compared to prior work or single-step models? Please clarify what is uniquely gained by studying the evolution of concepts across timesteps, and how this leads to improved understanding or control of diffusion models. \n\nThe authors intervene on the activations at different stages of the denoising process (t = 0.0, 0.5, 1.0), however it is not clear how many forward passes they perform.\n\n\n**Limited Model Scope and Generalizability**\nThe primary limitation is restriction to a single model (Stable Diffusion v1.4). This severely limits the generalizability of findings. Claims about \"diffusion models\" in the title and abstract are overstated given evaluation on only one architecture.\n\n\n**Interpretation Pipeline**\nThe authors argue that leveraging vision-language models to interpret learned SAE features is prone to hallucination and biases. This approach was used by Surkov et. al. As an alternative, they propose using other models such as pre-trained DINO backbones for segmentation and word2vec embeddings. However, those can also be subject to biases in the data, and the authors do not compare the two interpretation pipelines. In particular, they never measure hallucinations and biases of their approach, yet claim stronger results.\n\nAdditionally, the authors claim that their approach is more scalable. I would argue that vision language models are more general and scalable. Indeed, the authors need to create a multi-stage interpretability pipeine, whose components are not ablated at all, while using an end-to-end vision-language model would have fewer moving parts. Indeed, vision-language models do not rely on a precomputed list of concepts that must be represented with word2vec, a relatively old embedding model, which is likely to limit the expressivity of the intepretability pipeline.\n\n**Lack of Quantitative Evaluation for Key Claims**\nMost claims rely heavily on qualitative evidence. For instance, the claim that composition emerges in the first step (Figure 1) lacks rigorous quantitative validation. The IoU evaluation in Figure 3a provides some quantification but is limited and the reported IoU (0.26) for t=1.0 is quite low. In particular, I would not say that the final composition can be predicted \"surprisingly well\" (line 12).\n\n**Missing Baseline Comparisons**\nThe paper lacks comparison to other interpretability methods for diffusion models. The related work in Section 2 mentions several approaches, but no empirical or quantitative comparisons are provided to support the benefits of their approach.\n\n**Ad-hoc choices**\nSeveral evaluation choices appear arbitrary. The activation map binarization threshold of 0.1 (lines 235-236) lack justification. The intervention strength parameters in Table 2 vary dramatically without *any* principled selection.\n\n**Limited Analysis of Failure Cases**\nThe paper lacks discussion of when the approach fails. Section 4.3.2 mentions that some interventions \"cause visual distortions\" (lines 308-309) but does not analyze failures systematically.", "was_revised": true, "final_justification": "I am not convinced by the rebuttal of the authors. The lack of baseline, the fact that multiple design choices appear relatively ad-hoc, and the fact that the authors analyze a single model only still makes me believe that the paper should be rejected. Additionally, as stated in my original review, the presentation, especially the background, needs significant improvements.", "rebuttal": "We appreciate the reviewer’s time and thoughtful feedback. We're especially glad that the reviewer found the early emergence of the final composition of the scene surprising and noteworthy. Please find below our point-by-point responses to the reviewers’ comments.\n\n- **Re weakness 1:** We thank the reviewer for this helpful suggestion. We agree the background should better support readers unfamiliar with SAEs and diffusion. We will expand it and relocate definitions (e.g., classifier-free guidance, encoder/decoder) from the methods section as appropriate.\n- **Re weakness 2:** We thank the reviewer for raising this question. As state-of-the-art image diffusion pipelines (e.g., Stable Diffusion, Imagen, DALL·E) rely on a multi-step denoising process to synthesize images, we believe it is crucial to understand how semantic information emerges and evolves over these steps. Our claim that temporal analysis is critical stems from a broader goal: to demystify why the diffusion process is so effective as a generative modeling framework. Specifically, breaking down the mapping from noise to natural images into a sequence of intermediate steps is a deliberate design choice. Hence, understanding what is gained from this decomposition can help answer fundamental questions: Are there early commitments to layout or object identity? When does texture or fine detail emerge? Which steps are most causally influential in determining the final output?\nBy analyzing the time-evolution of interpretable concepts, our work sheds light on these dynamics. Such insights cannot be recovered from single-step or few-step diffusion models that bypass these. We believe such understanding is essential not only for interpreting current models, but also for guiding the development of more efficient, controllable, or interpretable generative systems. \n  We will revise the manuscript to better articulate this motivation and clearly differentiate our approach from prior work focused on single-step analyses.\n\n  Regarding the number of forwards steps: we collect the activations across 50 DDIM steps. We will clarify this in the revised manuscript.\n- **Re weakness 3:** Thank you for pointing this out. Collecting activations for millions of text prompts is costly time and storage wise. We did our best to perform a comprehensive set of experiments on a typical and popular architecture and provide the first SAE based temporal analysis of the evolution of interpretable concepts. In future work, we are interested in exploring further architectures, including Diffusion Transformers, where we expect our causal interventions to be even more effective. \n- **Re weakness 4:** We would like to first clarify that we do not report stronger results than the existing work that utilize vision-language models. In fact, our work is aimed at analyzing the time-evolution of concepts during the diffusion process, and is not a particular method where concrete numerical results of performance can be reported. We acknowledge that vision-language models can be also leveraged for concept discovery, however our design choice is to opt for an alternative approach due to the differences highlighted in our work. Replacing the concept discovery pipeline in our framework with vision-language models is not straightforward, as it is unclear how to build the concept dictionary in this case, and thus it is not a simple ablation study we could perform. With respect to the usage of Word2Vec embeddings: we agree with the reviewer that richer semantic embeddings could be used. A key benefit of our pipeline is its modularity: we can replace individual components of the annotation pipeline, or the semantic embeddings used to generate the concept dictionary. We used Word2Vec embeddings for the sake of simplicity. In fact, we believe that using embeddings from LLMs would be an interesting direction to further improve the granularity of our analysis. \n- **Re weakness 5:** We understand the reviewers’ concerns. We conducted two sets of experiments to bolster our claims quantitatively. First, we run experiments to quantitatively measure the purity and separation of the concepts that we discover and how it temporally evolves across diffusion iterates. We observe that as we progress along the reverse diffusion process, concepts in the dictionary cluster around the mean better (increased concept cohesion) and that the concepts are better separated (more distinct). We refer the reader to our response to “question 1 and weakness 1” of Reviewer HyPM for the full experimental details and results table.\n\n  Next, to evaluate the effectiveness of our intervention method, we introduce a proxy metric based on CLIP similarity. Specifically, we compute the CLIP distance between a generated image and the mean CLIP embedding of the target concept (obtained by averaging embeddings of all associated dictionary terms). An edit is considered successful if this similarity increases after intervention. We conducted this experiment on 100 prompts from the validation set and applied our editing method at three timesteps (early, middle, late), focusing on concept indices corresponding to cartoon, sea & sand, and painting (as shown in Figures 4, 5, and 6). \n\n  Results are as follows:\n\n  | Timestep | Concept ID | Before Edit | After Edit | Δ (After - Before) | Edit Success |\n  |----------|------------|-------------|------------|---------------------|---------------|\n  | t = 0.0  | 1722       | 0.198       | 0.206      | +0.008              | 0.82          |\n  |          | 2349       | 0.190       | 0.193      | +0.003              | 0.69          |\n  |          | 3593       | 0.198       | 0.204      | +0.006              | 0.79          |\n  | **Average** |            | **0.195**   | **0.201**  | **+0.006**          | **0.77**      |\n  | t = 0.5  | 1722       | 0.198       | 0.222      | +0.024              | 0.97          |\n  |          | 524        | 0.190       | 0.220      | +0.030              | 0.95          |\n  |          | 2137       | 0.198       | 0.219      | +0.021              | 0.93          |\n  | **Average** |            | **0.195**   | **0.220**  | **+0.025**          | **0.95**      |\n  | t = 1.0  | 1722       | 0.198       | 0.211      | +0.013              | 0.84          |\n  |          | 2366       | 0.190       | 0.193      | +0.003              | 0.56          |\n  |          | 2929       | 0.198       | 0.211      | +0.013              | 0.78          |\n  | **Average** |            | **0.195**   | **0.205**  | **+0.010**          | **0.73**      |\n\n  We find that edits are most successful at the middle timestep (t = 0.5), with a 95% success rate and the highest average ΔCLIP similarity (+0.025), compared to 77% (+0.006) and 73% (+0.010) at early and late stages, respectively. This aligns well with our qualitative observations and supports our main insight: intervening during the middle stages of diffusion offers the most effective control over image style while preserving content. \n  \n  Regarding the IoU scores at $t = 0$, an IoU of $0.26$ is not particularly high for a dedicated segmentation model, however our SAE has not been trained on any objective even remotely related to image segmentation, and thus the results are non-trivial. In fact, some zero-shot segmentation methods report comparable scores (albeit on a different dataset) [1]. We view this not as a shortcoming of our method (as it is not a segmentation model), but rather as an insight into the diffusion process itself: while a coarse layout (hence fairly low IoU) can already be predicted at $t = 1.0$ (also depicted in Figure 1), the final image composition is not yet fully determined at this early stage. \n\n  [1] Lüddecke, Timo, and Alexander Ecker. \"Image segmentation using text and image prompts.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n- **Re weakness 6:** Related work in Section 2 overviews some prior works in a similar spirit to ours, however none of them are directly comparable to ours due to the lack of focus on time-dependency. Furthermore, we would like to emphasize again that we do not present this work as a competitive controlled generation framework, but as a tool to better understand diffusion as a phenomenon. Our aim is to demystify the evolution of concepts. Being able to causally intervene in an interpretable way is a side-effect of the framework.\n- **Re weakness 7:** We pick the binarization threshold of 0.1 to filter out low-magnitude, noisy activations that we observe on most samples. We haven’t tuned this parameter extensively, however it is possible that further tuning could further improve the purity and localization of discovered concepts. \n \n  We determine the intervention strengths for both global and spatially targeted interventions by qualitatively inspecting generated images across a range of values (see Figure 6). The higher variance in the spatial intervention strengths reported in Table 2 likely stems from slight differences in normalization. For example, using strengths significantly above 400 (e.g., 1000 or 4000) during mid-stage spatial interventions often introduced visual artifacts.\n- **Re weakness 8:** We do mention two key limitations of the current work: first the discovered concepts and their spatial resolution is upper bounded by the accuracy/segmentation capability of the zero-shot pipeline we use. In fact, we visually observe that the image segmentation predicted from our concepts close to t=1.0 are more accurate than the segmentation maps provided by the zero-shot pipeline! Second, due to skip connections in the architecture, there is a “leakage” effect when attempting to intervene on specific concepts. We believe that this can be addressed by exploring further architectures such as Diffusion Transformers.\n\nWe appreciate your time and are happy to engage further should any questions remain during the discussion phase."}]

Author rebuttal:

Meta-review: The paper presents an interesting finding related to diffusion models with SAEs:

> even before the first reverse diffusion step is completed, the final composition of the scene can be predicted surprisingly well by looking at the spatial distribution of activated concepts

This is an important finding, potentially going against what various "guidance" of denosing diffusion models have been trying to achieve. These findings, however, have been related to recent work's findings, such as the one in (https://arxiv.org/abs/2411.18810), which have not been discussed. Other reviewers have also pointed out important relevant work that must be discussed. 

Still, the work is timely, and the findings show potentially important implications for a large body of generative modeling work. The AC thus weighs this strongly and recommends accepting the paper, but **urges** authors to enhance their camera-ready according to the dense feedback from the reviewers.

The ratings for this paper diverged, with a single reviewer, who, with all respect, focused on an aspect of the paper and decided to put a strong rating. The reviewer did not communicate with the AC or the fellow reviewers on their position either. Thus, despite the strong endorsement from another reviewer, this paper ended up being caught by an average rating criterion, which is known to be susceptible to outliers. 

The paper's methodology makes sense, is technically sound, and the novel findings that the paper provides should be of wide interest to the community. I do not believe anything less is deserving.

Additionally, conferences are where ideas are shared and communicated. It is to facilitate research, acting as if it's an event where researchers' neurons are connected beyond their own brains. The reviewing process should encourage such, and put more value on "is the idea worth sharing", and should not be a grading session. This paper clearly stands out in terms of value regarding the ideas presented in the paper.
Decision: Accept (spotlight)