One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models
Authors: Viacheslav Surkov
Chris Wendler
Antonio Mari
Mikhail Terekhov
Justin Deschenaux
Robert West
Caglar Gulcehre
David Bau
Venue: NeurIPS 2025 poster ()
Keywords: sparse autoencoders
SAE
unet
diffusion
mechanistic interpretability
text to image interpretability
activation engineering
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We analyze SDXL Turbo using SAEs and feature sharing across denoising steps and find specialization among the transformer blocks.

Abstract: For large language models (LLMs), sparse autoencoders (SAEs) have been shown to decompose intermediate representations that often are not interpretable directly into sparse sums of interpretable features, facilitating better control and subsequent analysis. However, similar analyses and approaches have been lacking for text-to-image models. We investigate the possibility of using SAEs to learn interpretable features for SDXL Turbo, a few-step text-to-image diffusion model. To this end, we train SAEs on the updates performed by transformer blocks within SDXL Turbo's denoising U-net in its 1-step setting. Interestingly, we find that they generalize to 4-step SDXL Turbo and even to the multi-step SDXL base model (i.e., a different model) without additional training. In addition, we show that their learned features are interpretable, causally influence the generation process, and reveal specialization among the blocks. We do so by creating RIEBench, a representation-based image editing benchmark, for editing images while they are generated by turning on and off individual SAE features. This allows us to track which transformer blocks' features are the most impactful depending on the edit category. Our work is the first investigation of SAEs for interpretability in text-to-image diffusion models and our results establish SAEs as a promising approach for understanding and manipulating the internal mechanisms of text-to-image models.

One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models
Viacheslav Surkov*.
Chris Wendler* Robert West
Antonio Mari!
Mikhail Terekhov David Bau²
Justin Deschenaux
Caglar Gulcehre'
LEPFL
"Notheesttern University
equul contribution
SOXL Turbo stop
SOXL Turbo steps
SDXL Base 25 steps
Original
Flux-schnell
Original
Figure 1: Enabling features learned by sparse autoencoders results in interpretable changes SDXL Turbo's, s. SDXL's, and Plux-schnell's image generation processes. The image captions correspond to feature codes comprised of transformer block name and feature index.
Abstract
For large language models (LLMs), sparse autoencoders (SAEs) have been shown decompose intermediate representations that often into sparse sums of interpretable features, facilitating better control and subsequent analysis. However, similar analyses and approaches have been lacking for text-to- image models. We investigate the possibility of using SAES to learn interpretable features for SDXL Turbo, few-step text-to-image diffusion model. this end, we train SAEs on the updates performed by transformer blocks within SDXL Turbo's denoising U-net in its 1-step setting. Interestingly, we find that they generalize to 4-step SDXL Turbo and even to the multi-step SDXL base model (i.e., a different model) without additional training. In addition, we show that their learned features are interpretable, causally influence the generation process, and reveal specialization among the blocks. We do so by creating RIEBench, a representation- based image editing benchmark, for editing images while they generated by turning on and off individual SAE features. This allows us to track which transformer blocks' features the most impactful depending the edit category. Our work is the first investigation of SABS for interpretability in text-to- image diffusion models and our results establish SAEs as promising approach for understanding and manipulating the internal mechanisms of text-to-image models.
interpretable directly
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Text-to-image generation is rapidly evolving field. The DALL-E model first captured public interest 46]. combining learned visual vocabularies with sequence modeling to produce high-quality images based on user input prompts. Today's best texl-to-image models are largely based text-conditioned diffusion models (50
This can be partially attributed to the stable training dynamics of diffusion models, which makes them easier to scale than previous approaches such as generative adversarial neural networks [ As result, they can be trained on internet scale image-text datasets like LAION-5B
51
52
[33] and learn to generate photorealistic images from text.
However, the underlying logic of the neural networks that enable the text-to-image pipelines we have today, due to their black-box nature, is not well understood. Unfortunately, this lack of interpretability is typical in the deep learning field. For example, advances in image recognition (28] and language
modeling [15 3
come mainly from scaling models
(2I, rather than from improved understanding
of their internals. Recently, the emerging field of mechanistic interpretability has sought to alleviate
this limitation by reverse engincering visual models (37] and transformer- LLMs 45 At the same time, diffusion models have remained underexplored.
This work focuses SDXL Turbo, a recent open-source few-step text-to-image diffusion model. We import methods from a toolbox originally developed for language models, which allows inspection of the intermediate results of the forward pass 8 20 [ [5]. Moreover, some of these methods even enable reverse engineering of the entire task-specific subnets B5I. In particular, sparse autoencoders (SAEs) 62
are considered breakthrough in interpretability for LLMs. They have been shown to decompose intermediate representations of the L.LM forward pass often difficult to interpret due to polysemanticiry into sparse sums of interpretable and monosemantic features. These features are learned in an unsupervised way, can be automatically annotated using LLMs 7 and facilitate subsequent analysis, for example, circuit extraction [35]
1.1 Contributions
In this work, we investigate whether we can use SAFs to localize semantics to vector representations specific layers of SDXL. Turbo, recent open-source few-step text-to-image diffusion model.
SDLens. To facilitate our analysis, we developed a library called SDLens that allows us to cache and manipulate intermediate results of SDXL Turbo's 's forward pass. We use our library to create a dataset of SDXL Turbo's intermediate feature maps of several transformer blocks inside SDXL We then use these feature maps to train multiple SAEs for each transformer block. We source the code of SDLens
Turbo's U-net on 1.5M MAION-COCO prompts [33 5el
SAEPaint. To qualitatively assess the role and the impact of features learned at different transformer blocks in SDXI. Turbo we create feature visualization techniques based on examples in which the features are highly active and on various interventions. To interactively perform interventions, e.g.. adding feature on a spatial region (at specific transformer block) or subtracting while generating an image, we build an app SAEPaint (Appendix Ab. Our qualitative analysis (Appendix G and Appendix HD suggests that the blocks specialize into a "composition," a "detail," and a "style" block
RIEBench. In order to quantify the strong causal effects that we qualitatively observe when inter- acting with SDXL Turbo and SDXL via SAEPaint, we create a new representation-bun based image editing benchmark (RIEBench), for our setting of turning on and off features as we generate new images. We do so by leveraging the nine edit categories including prompts from PIEBench 26 and combining them with grounded SAM2 49 to compute semantic segmentations, which allows for the selection of the constituting features. We find that SAE features allow for fine-grained transfer of visual features across different denoising processes matching neuron baselines while requiring multiple orders of magnitudes less features.
By analyzing the features selected with regard to the edit category we quantify the specialization of SDXL Turbo's 's transformer blocks that we also observe when interacting with SDXL through SAE-
A phenomenon where single neuron feature encodes multiple, unrelated concepts
2 "Our project page links to mentioned repositories and resources wawtrrcssstttpt//// httpp:///dtttllliaeeee/ unbox. epfl. fofl..ch ch
Our "composition" and "style" blocks have been already known in the community (57.

---

Paint. In particular, the "detail" block was most effective for adding/deleting'hanninn objects/details and the "style" block for changing color/background and style.
Generalization across steps and architectures. Despite training on the one-step process we find that learned features without requiring additional training generalize to the SDXI.Turbo's four-step process and even vanilla SDXL's multi-step process. We find that one-step denoising is enough for training meaningful SAE features applicable to multi-step denoising. Our one-step SAEs not only retain much of their reconstruction quality but also their learned features retain their semantic meaning and causal influence on the multi-step stet generation (see Fig. or Appendix E). We quantify this by analyzing reconstruction performance and feature overlap across multiple denoising steps.
Additionally, we train SAEs that can be used to manipulate the generative process of FLUX Schnell 30 (also Fig. [ Again we observe that training on the one step generation process enough. We consider this erucial result because it demonstrates that meaningful, interpretable features can be extracted with significantly lower computational resources by training on the distilled model and then effectively deployed understand and edit more powerful, multi-step models.
Thus, we show that SAEs learn interpretable features that causally the image generation process. By open-soureing our library and SAEs, we lay the foundation for further research in this area.
2 Background
2.1 Sparse Autoencoders
Let h(x)) E Rd be intermediate result of a forward pass of a neural network on input In fully connected neural network, h(z) could correspond to a vector of neuron activations. transformers, which are neural network architectures that combine attention with fully connected layers and residual connections, hxx) could either refer to the content of the residual stream after a layer, an update to the residual stream by layer, or a vector of neuron activations within fully connected layer.
has been shown 62 that in many neural networks, especially L.MMs, intermediate represen- tations can be well approximated by sparse sums of n E N learned feature vectors, i.e.,
h(T) (z)Lf
where (1))) are the input-dependent coefficients, most of which equal and fj..... Lnz Rd is a learned dictionary of feature vectors. Importantly, the features are usually interpretable.
Sparse autoencoders. To implement the sparse decomposition from equation I the vector containing the n coefficients of the sparse sum, parameterized by single linear layer followed by an activation function, called the encoder,
ENC(h) a (W-BC bpre) + bas);
which h E Rdd is the latent that we aim to decompose, ar.)) is an activation function, W Ty xd is a learnable weight matrix and biee and bust are learnable bias terms. We omitted the dependencies h(x) and s = sh)), which are clear from the context.
Similarly, the learnable features are parametrized by single linear layer called decoder,
h' DEC(s) DDDEC + bpue.
in which W/DEC_ ... features and bpre is a learnable bias term.
(..,)) E dxdyy a learnable matrix. Its columns take the role of learnable
2.2 Few-Step Diffusion Models: SDXL Turbo
SDXL Turbo
52
is distilled version of Stable Diffusion XL 4 a powerful latent diffusion model.
SDXL Turbo allows high-quality sampling in as few as 1-4 steps. employs denoising network
implemented using U-net similar to [50]. For an architecture diagram consult Appendix D Fig. 10
An extended version of this section, including training details, is in Appendix K and Appendix L
3

---

The U-net consists of a down- down-sampling path, bottleneck, and an up-sampling path, each comprising one or more U-net blocks connected via up- and down-samplers. U-net blocks are built from residual networks, and some blocks incorporate multiple cross-attention transformer blocks while others do not. We refer to these transformer blocks by their short names (e-g.. (lww..2.1). Each transformer block is composed of several basic transformer layers, including self-attention, cross-attention, and MLP layers. Importantly, the text conditioning is achieved via cross-attention to text embeddings performed by 11 transformer blocks embedded in the down-, up-sampling paths, and the bottleneck. An architecture diagram displaying the relevant blocks can be found in additional material.
3 Sparse Autoencoders for SDXL Turbo
With the necessary definitions hand, in this section we show way to apply SAEs SDXL Turbo.
Where to apply the SAEs. Since we want to localize blocks' responsibilitiies, we apply SAEs to the updates performed by the transformer blocks containing the cross-attention layers responsible for incorporating the text prompt (for details consider Appendix Dp. Fach of these blocks consists of multiple transformer layers, which attend to all spatial locations (self-attention) and the text prompt embeddings (cross-attention).
Formally, the 9th cross-attention transformer block updates its inputs in the following way
D(g = DIO + "I c)g.
(4)
in which Di Diejout E Rxxxxx denote the residual stream before and after application of the b-th cross-attention transformer block respectively. The transformer block itself calculates the function T(e] Rxxuxd - Rhxwxd Note that we omitted the dependence on input noise 2n and
text embedding e for both DI0)" (zc, c) and D[ (ze, c).
We train SAEs on the residual updates [[ c).g E R denoted by
ADIA, = T!D(D c). DIA DI4
(5)
That is, we jointly train one encoder [NC[ and decoder DEC pair per transformer block & and share it over all spatial locations j.j. We do this for the 4 (out of 11) transformer blocks that we found have the highest impact on the generation, namely, down. 2. 1, mid. O, up.0.0 0.0 and up.0.1.
For the sake of simplicity, we omit the transformer block index l in the remainder of the paper.
Feature maps. We refer AD Rxx as dense feature map and applying ENC to all image locations results in the sparse fearue map hxxxxx with entries Sij = END().
We refer to the feature map of the pth learned feature using S E R This feature map S contains the spatial activations of the pth learned feature. Its associated feature vector fp E R is Rdxa,, Using this notation, we can represent
column in the decoder matrix W-DBC_ = (fil.-. ... |F,,) each element of the dense feature map as a sparse sum
ny
D
Stfo with St = 0 for most p E (l....,
(6)
p=1
Training. In order to train an SAE for a transformer block, we collected dense feature maps A from SDXL Turbo one-step generations on 1.5M prompts from the LAION-COCK 54 Each feature map has dimensions of 16 x 16, resulting in training dataset of 384M dense feature vectors per transformer block. For the SAE training process, we followed the methodology described [1)], using the TopK activation function and an auxiliary loss to handle dead features. For more details on the SAE training and for training metrics, consider the supplementary material. In order find good hyperparaneters, we perform a line search for an optimal sparsity parameter ls while keeping ny fixed 5120 (expansion factor 4) and a line search for for n, where we keep fixed at 10. As can be seen in Fig. 2 the explained variance strictly increases when increasing is and similar trend holds for the expansion factor, proportional to the number of features ny.
We provide an architecture diagram in Appendix D Fig. 10

---

Block
Block
down.2.1
down.2.1
up.0.1
0.9
up.0.1
Sppained Variance
up.0.0
(plained Variance
up.0.0
mid 0
mid 0
0.8
0.8
0.7
0.7
0.6
0.5
16
10
20
80
160
Expansion Factor
k
Figure 2: The explained variance increases with both the expansion factor and k. The left shows the dynamics of explained variance for SAEs with varying expansion factors and fixed k 10, while the right panel illustrates the same for varying k values with a fixed expansion factor of 4.
Generalization study. We trained the SABs the 1-step generation process of SDXL Turbo. However, most diffusion models (including SDXL. Turbo) require multiple denoising steps to create high quality images. To assess our SAEs potential for application across multiple denoising steps, we compute their explained variance across the 100 randomly generated images for SDXL. Turbo's 4-step setting and the vanilla SDXL base model's multi-step setting (see Fig. 3 left) R
One can see that the explained variance remains high across the entire denoising process, which suggests that our SAEs also applicable for the 4-step generation process and even the base model's 20-step process.
This suggests that our feature intervention (formal definition below) should causally influence the generated image in multi-step diffusion in the same way it does in the 1-step process where it was trained. This is visualized in the first three rows of Fig. T for example, adding the 2301st feature vector to the forward pass consistently makes the output look more "evil." The 4th row of Fig. D shows steering examples for Flux-schnell (30) as well, using features obtained with an SAE trained on 1-step activations of layer 18. This shows the generalizability of our approach to recent diffusiin-ttrasfoormer
[Л]. Additional examples of FLUX interventions are presented in Appendix C Further examples of vanilla SDXL interventions and the effect of intervening only on
subsets of the denoising steps are shown in Appendix E
On the right of Fig. we visualize how the SAE features change across denoising steps. We do so by computing the cosine similarity between the SAE coefficients of adjacent timesteps. Interestingly, the set of features stabilizes rapidly and subsequent denoising steps' features stay highly overlapping for most of the denoising process. We therefore proceed by using SDXI. Turbo's 's 4-step process from now.
Feature interventions. We design interventions to turn on and off the p-th feature. Specifically, we achieve this by adding or subtracting across spatial locations i, weighted by hgxxw
AD', AD + ArpEp:
in which A Dij is the update performed by the transformer block before the intervention and AD after the intervention, and f, is the p-th learned feature vector. Our examples in Fig. are obtained by drawing binary masks in SAEPaint and multiplying them with an scalar specifying the intervention strength, i.e. A = aM in which E R and M E {0, 1
Note that naive application of equation is sufficient when using SDXL Turbo with default settings, that without classifier-free guidance. However, with classifier-free guidance (which is enabled in the SDXL base model), we add the features only during the next-conditioned forward pass and subtract the same features during the unconditional one. When adding the same features during both forward passes, the features' effects inhibit each other (see Appendix B Fig. 7
4 Analysis of SDXL Turbo
In this section we leverage our SAES to better understand how SDXL Turbo generates images.
This results in 25600 latents for SDXL Turbo and 102400 latents for SDXL.
5

---

1.0
0.8
Explained Variance
SAE Cosine Similarity
0.6
Block
0.6
SOXL-Turbo: down
SDXL-Turbo:
0.4
SDXL Turbo: up.0.0
up pp.0.1
SDXI-Rase: down-2.1
0.2
mid.0
SDXL Base: up.0.0
SDXL Base: up.0.1
0%
10% 20% 30%
40% 50% 60% 70% 80% 90% 100% Denoising Progress
0% 10% 20%
30%
40% 50% 60% 70% 80% 90% 100% Denolsing Progress
Figure 3: For our best SAEs k 160, n 5120, we compute how their explained variance changes across denoising steps in the multi-step setting (for which they were not specifically trained) on the left. On the right, compute SAE feature overlap by computing cosine similarities of the sparse ny dimensional SAE feature vectors. The SAE feature overlap is remarkably high across timesteps. The plot ends early because compute overlaps between t and 1 respectively.
4.1 RIEBench: Representation- Based Image Editing Benchmark
source "a leopard on rock"
target "a bird on rock"
result
Figure 4: We perform feature transfer experiment, computing semantic image segmentation masks using grounded SAM2 49 for a source and target image. We collect features inside the mask of the source image at SDXL Turbo's 's intermediate layers and insert them (via addition) during a new forward pass with the same prompt as the target one, We also subtract features collected target forward pass. The resulting image is a blend of the two forward passes.
In order to quantify the causal impact of SAE features on SDXL Turbo's generation, we introduce RIEBench, representation-bu based image editing benchmark. To generate RIEBench samples, we first create two parallel forward passes (source and target forward pass) that receive the same random input noise and mostly parallel prompts except that they are differing in one visual aspect. Next, we transfer features from the source forward pass into the target forward pass. E.g., in Fig. 4 the prompts are leopard standing on top of rock" and bird standing on top rock." order to do so, we select spatial locations using grounded SAM2 [9 masks, obtained by prompting grounded SAM2 with "leopard" and "bird." Finally, we select a subset of the features that fall inside these masks in four blocks and create new forward pass with the target prompt in which subtract some of the target features and add some of the source features. This results in the new image on the right that depicts visual features of both the leopard and the bird.
We sample our source and target prompts from PIEBench 26 a prompt- image editing benchmark for diffusion inversion methods. PIEBench contains the "original prompts" (target of the transfer) and the "edit prompts" (souree the transfer) for different edit categories. We omit the examples from the random category and implement specialized interventions for the other edit categories Appendix F). For each edit category, we manually create grounded SAM2 prompts for the first 50 examples. For each of these examples, we fix random seed, generate images for source

---

Neurons
Steering
12800
25600
51200
102400 204R00
160
120
640
s.0.5
A
A
WHO.5
sw0.25
# 5=1.00
5-1.50
5-1.5
s-155
$02.00
Change object
Add object
Delete object
*
0.04
better
0.03
CLIP CLIP Increase
0.02
0.01
0.00
0.2
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Change content
Change pose
Change color
*
*
*
CLIP Increase
0.02
0.01
0.00
0.2
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
04
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Change material
Change background
Change style
*
0.04
CLP CL Increase
0.03
0.02
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4 0.5
0.6
0.7
LPIPS
LPIPS
LP PS
Figure 5: LPIPS with original image (x-axis) versus increase in CLIP similarity with the edit prompt (y-axis) from feature-transfer interventions using SAE features, neurons, and activations across nine edit categories from PIEBench [26]. Features from the edit prompt (within SAM2 segmentation mask) are transferred into the original prompt's forward pass. SAE experiments are blue, neurons red, and steering green. We plotted one line per number of features/neurons transported with increasing opacity proportional to the number. For SAEs we transport 40, 80, 160, 320, 640 features at time and for neurons 12,800 25,600 51,200 102,400 204,800. For SAEs/neurons we use strengths 0.5, 1.0, 1.5 and for steering 0.25, 0.5, 1.0, 1.5, 2.0.
and target prompt, and manually inspect whether all relevant visual features are present and whether they are selected by the grounded SAM2 masks and only those
Metrics. From interpretability standpoint, our benchmark jointly measures features' sensitivity, specificity. and causality. Visual features of the source forward pass can only be detected if their corresponding representations (c.g., SAE features) are sensitive. On the other hand, editing the gener- ated image to be closer to the source image via interpretable visual changes requires both specificity and causality of the features. When feature is active during the generation, should ideally result in corresponding interpretable visual feature and minimally interfere with the remaining ones. We quantify this aspect by transferring varying numbers of features between two parallel forward passes and measuring the LPIPS 64 distance between the intervened image and the original image and the increase in CLIP by transporting a varying number of features and increasing/decreasti their strength, it should be possible to smoothly interpolate between the relevant aspects of the source and the target image.
E similarity of the resulting intervened image with the edit prompt. Ideally,
7 After filtering. the edit categories and the corresponding numbers of selected examples were: change object (36), add object (27), delete object (43), change content (21), change pose (20), change color (30), change material (26). change background (46), change style (47).
7

---

Methods. We compare our best SAEs (k- = 160 and n, = 5120) with neurons of their corresponding transformer blocks. Since there are 10 MLP layers within each block that we decompose, there are 51200 neurons per block in total. Additionally, we create a simple method similar to activation steer- ing K3 which adds the AD within the mask from the source forward pass to the target forward pass and subtracts the A D within the target forward pass' mask from the target forward pass. To mimic interventions that can be performed in SAEPaint we aggregate feature values/neuronssaccivvtiin over spatial locations before adding them to masked area. Furthermore, all of our interventions are timestep step specific, i.e., the updates performed to the masked areas vary across denoising steps.
F Feature selection. For our feature selection step, we first aggregate the sparse feature maps by taking the mean over timesteps. Let S[ E Axx denote a sparse feature map at layer & and timestep t, we use $1 = + Ei= St Then to determine which features to transport, we rank each feature P according to score y[e], that quantifies the difference in the magnitude of p across the source and target, aggregated over the respective grounded SAM2 masks and normalized by the sum of all feature magnitudes (c.f. Cywiński and Deja (22::
8 $4
(8)
- s|1
Ep
where s|(pp= J E Ei Mrr S161;, and s|[]>" i Eijem StG, To select features
among multiple layers we simply concatenate the layer specific score vectors > E RR", i.e., ? = 7/down.2.1 7[mid.] [o]p.000 -7[up...] E I H where denotes concatenation, before ranking.
For the neurons, we perform a similar ranking (see Appendix [ for details).
Results. We show the results in Fig. 5 Quantitatively, in most categories there are no significant differences between the considered intervention types. Given an LPIPS budget, they all achieve similar increases in CLIP similarity (top left is better) in all categories except change object in which neurons dominate for higher LPIPS values, and, change content, change color, change material and change style in which both steering and SAFs are significantly more effective than neurons. While neurons also cover a good range of L.PIS distances / CLIP similarities, SAEs come with the benefit of requiring several orders of magnitude fewer features to do so. Interestingly, the simple steering baseline that works directly with the activations is highly effective as well. It's worth noting that when sufficiently increasing the number of transported features the SAE intervention approaches the steering intervention up to the reconstruction error introduced by the SAE. On the change pose task none of the considered methods works well. The change pose tasks is hard to achieve using our current RIEBench setup that in essence consists of adding constant update vectors to a masked areas.
We also provide qualitative examples and the same evaluation for FLUX Schnell in Appendix F
4.2 Specialization Among Blocks
Thanks to our automated feature extraction with grounded SAM2 and our importance ranking, we can investigate which transformer blocks were relevant in terms of the number of features selected from that block depending on the task (see Fig. 5). It stands out that mid. does not have a high causal impact on the generation. Similarly, down 2. does not seem to get selected much for the edits considered. This surprising given its strong impact in our qualitative experiments using our app (c.g.. Fig. #2301, #4998, #3912). The 0p.0.1 block is most relevant for changing color, material, background, and style editing categories. For the other categories, up.0.0 0. is the most impactful. We also observed qualitatively that up. 0.O is good for editing local details as long as the context is relevant (e.g., [ #1911, which we think happens in this setting due to the paired prompts. We have an extended analysis of the roles of the blocks in Appendix Gland Appendix I
5 Related Work
Layer specialization in diffusion models. Similar to our findings on the roles of SDXL Turbo's 's transformer blocks, (59 65 observe specializations among the layers and denoising steps of text-to-image diffusion models as well. Voynov et al. [5 introduce layer-specific embeddings for the text conditioning and find that different sets of layers are more effective for influencing the generation of specific concepts such as styles objects. For a selected set of image attributes (c.g., color, object,
8

---

down.2.1
mid
Prompt
Original Pronpt
0.5
0.3
0.5
0.4
0.4
0.3
0.3
0.1
uodn 0000
0.4
0.1
Ohange object
0.0
and ooject
dalete object
change content
change proto
change color
change material
change background
change style
change stpeect
0.0
add object
delete abject
change content
change pase
change color
change naterial
charge bachground
change style
Figure 6: We count how often feature of a block selected for each task category.
layout, style), Agarwal et al. I analyze which attributes are captured in which timestamp and layer. They also find that subset of these attributes is often captured in the same layer and across the same denoising step. Zhang et al. (65] observe that during the denoising process of diffusion models first the layout forms (in early timestamps), then the content, and finally the material and style.
In contrast to these prior works, our approach takes a fundamentally different perspective and does not rely on either handcrafted attributes prompts. Instead, we introduce a new lens for analyzing SDXL Turbo's transformer blocks, which reveals specialization among the blocks as well. Interestingly, our findings on SDXL Turbo, which is distilled, few-step diffusion model parallel 655 's observations, which identify that composition precede material and style. In SDXL Turbo, this progression occurs across the layers instead of the denoising timestamps.
Analyzing the latent space of diffusion models. Kwon al. 24 show that diffusion models have semantically meaningful latent space. Park et al. [40] analyze the latent space of diffusion models using Riemannian geometry. et al. [3I and Dalva and Yanardag [ present sule-supe
methods for finding semantic directions. Similarly. Gandikota et al. 18] show that the attribute variations lie low-rank space by learning LoRA adapters [23 on top of pre-trained diffusion models. Brack et al. 4 and Wang et al. [60] demonstrate effective semantic vector algebraic operations in the latent space of DMs, as observed by Mikolov et al. 36 However, none of those
works train SAEs to interpret and control the latent space.
Mechanistic interpretability. Sparse autoencoders have recently been popularized by [5], in which they show that it is possible to learn interpretable features by decomposing neuron activations in MLPs 2-layer transformer language models. the same time, parallel work decomposed the elements of the residual stream [1]. which followed up on [55] To our knowledge, the first work that applied sparse autoencoders to transformer- based LLM was (62), which learned a joint dictionary for features of all layers. Recently, sparse autoencoders have gained much traction, and many have been trained even on state-of-the-an LLMs 19
58 32.. In addition, great tools are available for inspection [3 and automatic interpretation f of learned features.
35 have shown how to use SAE
features to facilitate automatic circuit discovery.
The studies most closely related to our work are 2. and 14 Ismail et al. [24 apply concept bottleneck methods (27 that decompose latent concepts into vectors of interpretable concepts to generative image models, including diffusion models. Unlike the SAES that we train, this method requires labeled concept data. Daujotas [44] decomposes CLIP KEl SAES and use them for conditional image generation with a diffusion model called Kandinsky [47] Importantly, using SAE features, they are able to manipulate the image generation process in interpretable ways. In contrast, in our work, we train SAES on intermediate representations of the forward pass of SDXL Turbo. Consequently, we can interpret and manipulate SDXL. Turbo's 's forward pass finer granularity, e.g., by intervening on specific transformer blocks and spatial positions.
vision embeddings using

---

Another closely related work to ours is 22, in which neurons in generative adversarial neural networks are interpreted and manipulated. The interventions in 2 are similar to ours, but on neurons instead of sparse features. To identify neurons for a semantic concept,
[ require semantic segmentation maps.
6 Conclusion and Discussion
We trained SAEs on SDXL Turbo's opaque intermediate representations. This study is the first in the academic literature to mechanistically interpret the intermediate representations of modern text-to-image model. Our findings demonstrate that SAEs can extract interpretable features and have significant causal effect on the generated images. Importantly, the learned features provide insights into SDXL Turbo's forward pass, revealing that transformer blocks fulfill specific and varying roles in the generation process. In particular, our results clarify the functions of down. 2. 1, up and up. O. 1. However, the role of mid. O remains less defined; it seems to encode more abstract information and interventions are less effective.
We follow up with a discussion of the results and their implications for future research. Based on our observations, we suggest a preliminary hypothesis about SDXL Turbo's 's generation process: down.2. 2.1 decides on top-level composition, mid. O encodes low-level semantics, up.0. 0.0 adds details based the two above, and up- op.0.3. fills in color, texture, and style.
Our work focuses on analyzing SDXL Turbo's ' intermediate representations. As a relatively compact, few-step diffusion model with small number of naturally partitioned components, SDXL Turbo turned out to be convenient to analyze with SAES. However, the application of the proposed techniques to larger and more complex text-to-image diffusion models with alternative architectures represents promising direction for further research. We provide some motivational results on Flux-schnell 30 in the supplementary material. Additionally, we observe that SAE features learned on SDXL Turbo's one-step generation are applicable to 4-step and to vanilla SDXL multi-steps generations. We observe that for FLUX training on one step is enough as well
This fact suggests that our SDXL SAES are also applicable to most of the 8,700 SDXL adaptations available on huggingface The same should hold for our FLUX SAEs, which should generalize to most of the 36, 148 FLUX adaptations of on huggingface. This type of generalization is similar to what has been observed in the classic model merging literature [61] where averaging the weights of finetunes of the same base model has been found to improve model performance. Recently, this compatibility between different finetunes of the same base model has been rediscovered and expanded in weight-space learning, where probes that take model weights as inputs are trainable and generalize within a collection of finetunes of the same base model (but not across different seeds) [2].
Future directions. Analyzing larger diffusion models with higher number of diffusion steps would benefit from advanced interpretability techniques capable of capturing connections between its components and across denoising steps. For example, such techniques are explored in B55 34] Marks et al. [55 compute circuits showing how different layers and attention heads wire together and Lindsey et al. 3e introduce cross-coders, a variation of SAES that allows to learn a shared set of features over latents corresponding to different layers.
Broader impact. This work explores the applicability of SAES on text-to-image models en- interpretable feature representations. As such models become
courages disentangled and increasingly powerful and widely adopted for image synthesis, they may be misused for generating deceptive content (c.g., deepfakes). Thus, understanding their internal representations is critical and improving interpretability can help mitigate such misuse by enabling researchers detect manipula- tion or unintended behaviors within these models. Overall, our work advances the development of interpretable, responsible, and numan-centered AI systems.
Limitations. Our current method focuses on individual blocks and may miss features that require combinations of multiple transformer blocks. The success rate and disentanglement quality of edits varies depending on the feature type and target image content. While we demonstrate local changes effectively (like skin color modifications). the identification and control more global and compositional features remains challenging. We focused on SDXL and FLUX, thus, generalization to other models requires further validation. The hyperparameter choices for SAE training impact the types and granularity of discovered features, requiring tuning.
'ased on subiikyaisaable diffusion base and fores: Law. of 7th of October 2025.
10

---

Acknowledgements
The authors thank Danila Zubko for the initial contribution, Peter Baylies, Rohit Gandikota, Rudy Gilman, Bartosz Cywińsski, Julian Minder, Imke Grabe, Niv Cohen, Gytis Daujotas, Tim R. Davidson and Alexander Sharipov for the valuable discussions and feedback. We also express our gratitude to EPFL RCP and IC clusters maintainers. Robert West's lab is partly supported by grants from the Swiss National Science Foundation (200021_185043, TMSGI2_211 211379), Swiss Data Science Center (P22 08). H2020 (952215), Microsoft, and Google. Caglar Gulcehre's lab is supported by nimble.ai. Chris Wendler is supported by NSF grant #2403304.
References
[1] Aishwarya Agarwal, Srikrishna Karanam, Tripti Shukla, and Balaji Vasan Srinivasan. An image is worth multiple words: Multi-attribute inversion for constrained text-to-image synthesis. arXiv preprint arXiv:2311.1190 2023.
[2] David Bau, Jun- Jun 7hu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenhaum, William T Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations, 2019.
[3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufci Guo, Improving image generation with better captions. Computer Science. https://ddn. openai. com/papers/dall-e-3 pdf. 2(3):8, 2023.
[4] Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing text-to-image models using semantic guidance. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances Neural Information Processing Systems, volume 36, pages 25365-25389. Curran Asso- ciales, Inc., 2023. URL
http://ppp.eeeoeee....erpeeeeees pdf
[5] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, and Adam Jermyn et al. Towards monose- manticity: Decomposing language models dictionary learning. Transformer Circuits, October 2023. URL
httpa://transforeer: circuits. pwbp http22////mmooeeaaatccc-eetattte
[6] Tom Brown. Language models are few-shot learners. arXiv preprint 1IXi:2005..44155 2020.
[7] Juang Caden, Paulo Gonçalo, Drori Jacob, and Belrose Nora. Open source automated interpretability for sparse autoencoder features, 2024. URL ftttp:://blog.eleetteee www.eeeeee....eaatttttttteeeeeee Accessed: 4024-09-27
[8] Haozhe Chen, Carl Vondrick, and Chengzhi Mao. Selfie: Self-interpretation of large language model embeddings. arXiv preprint arXiv:2403. 109499999999, 2024.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 2818-2829, 2023.
[10] M. Cimpoi, S. Maji. L. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
[11] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint (XXi::3309.08600 2023.
[12] Bartosz Cywiński and Kamil Deja. Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders, 2025. URL https://aaa/xiii http://////.iiiioooababbb////// absp//01b//5111... 18052
[13] Yusuf Dalva and Pinar Yanardag. Noisecir: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24209-2444 24218, June 2024.
[14] Gytis Daujotas. Interpreting and steering features in images. 2024. URL httpst/t://www hatpp::/www.eessvrrong com/ posts/ hhtpg//wworsBooekcccceeeeeeeeeee 0024-09-27
and steering features- in- images
Accessed:

---

[15] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv:1810.049/// 2018.
[16] Prafulla Dhariwal and Alex Nichol. Diffusion models heat gans image synthesis, 2021. URL https
//arxiv.org wtwpv//.rggbbbaa/eeaeeea 05233
[17] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hadfield-Dadds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits, September 2022. URL artpa://bransforrer-crrroo circcitictttt... pub/ pub/2022/toy 1tto0/2///oooooeelee_o_e mtt.mmmmeeeddexeeeeeeeee
html
[18]
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models, 2023. URL hattps://arxtti http//axxvvvvgg////// aba/ 2311
12092
[19] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autocncoders. arXiv preprint arXiv:2406.04094 2024.
[20] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscope: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06
2024.
[21] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, ct al. Training compute-optimal large language models. arXiv preprint ArXiv:2203. 15556, 2022.
[22] Eliahu Horwitz, Bar Cavia, Jonathan Kahana, and Yedid Hoshen. Learning on model weights using tree experts. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20468-20400 2025.
[23] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL WtRphhttss///axii org/ abs/2106.0095 09685
[24] Aya Abdelsalam Ismail, Julius Adebayo, Hector Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Concept bottleneck generative models. In The Twelfth International Conference on Learning Representations, 2023.
[25] William B Johnson, Joram Lindenstrauss, and Gideon Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal Mathematics, 54(2):129-138, 1986.
[26] Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion- based editing with lines code, 2023. URL httts:://tstiitii org.:iii.orrreeeeee org/abs//////. 2at.b//2110.110
[27] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338-5348. PMLR, 2020.
[28] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[29] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusson models already have a semantic latent space.
2023. URL
[30] Black Forest Labs. Flux.
htmm/baack-fooe
2024.
[31] Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. Self-discovering interpretable diffusion latent directions for responsible text-to-image generation. In Proceedings of the IERF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12006-12016. June 2024.
[32] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anea Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXin:2408.05147 2024.
[33] Johnny Lin and Joseph Bloom. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL https::///// http://wwwwwpoottttp./// org Software available from neuronpedia.org.
12

---

[34] Jack Lindsey. Adly Templeton. Jonathan Marcus, Thomas Conerly. Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits, October 2024. URL
https://transforrer- circurrrreee--rt-teeeee
httpa////roossooer//nneeeeee html
[35] Samuel Marks, Can Rager, Erie J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint MMXiv:2403.19647 2024.
[36] Tomas Mikolov, Kai Chen, Greg Corrddo, and Jeffrey Dean, Efficient estimation of word representations in vector space, 2013. URL httppst////axii..rreee// /tt.b/////...... 37881
[37] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview early vision in inceptionvl. Distill, 5(4):00002 002, 2020.
[38] OpenAI. Hello gpt-4o, 2024. URL 28.
Accessed: 2024-09-
[39] Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024. https://////a httttp::://axxi..rrr/bbbb/ htrg/abs/2112 066811
[40] Yong-Hyun Park. Mingi Kwon. Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Under- standing the latent space of diffusion models through the lens of riemannian geometry. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine. editors, vances in Neural Information Processing Systems, volume 36, pages 24129-24142. Curran Asso- ciates, Inc., 2023. URL https: htttp:://proceedinggs neur ips. wwtpcccpppeaacaecceee wwt..//sppppr///2////a... 4bf w217 0tt.0/00000771111001000110111000 Pttp://9900-PPPeP----e Paper- Conference.. pdf
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https
//axxiv.orz wwwpxiv.org/aaa///e wtwgg//////////a/a 09748
[42] Pablo Pernia,, Dominic Rampas, Mats L Richter, Christopher J Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. arXiv preprint arXiv:2306.0063) 2023.
[43] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna. and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
URL httpt://aaxxii org/ aba/ 2307 .01522
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[45] Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint 2tt::22007022644 2024.
[46] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning. pages 8821-8831. Pmlr, 2021.
[47] Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, llya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky: An improved text-to-image synthesis with image prior and latent diffusion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 286-295, 2023.
[48] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. 11 2019. URL URL https:://txxx:/ org. http://bs///9/.... 10084
[49] Tianhe Ren, Shilong Liu. Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng. Hao Zhang. Feng Jie Yang, Hongyang Li, Qing Jiang. and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024, URL /tapss///axiv...o org/ abs/2401. 14159
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- image synthesis with latent difffusion models, 2022. URL htttp:://arxiii org. abs /ttps///2....... 10752
13

---

[51] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton. Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusson models with deep language understanding. Advances in neural information processing systems, 09366499-64944 2022.
[52] Axel Sauer, Dominik Lorenz, Andreas Blaumaann. and Robin Rombach. Adversarial diffusion distillation. 2023. URL
17042
[53] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman. Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Saion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 2222247--5594,, 2022.
[54] Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, Romain Bcaumont, and Benjamin Trom. Laion coco: 600m synthetic captions from laion2b-en, 2022. URL https://laion a1/blog/ caion-coco/ Accessed: 2024-10-01.
[55] Lee Sharkey, Dan Braun, and beren. Interim research report: Taking features out of superposition with sparse autoencoders, 2022. URL https://www http:///ww...oeoooo////////////p
report- - tawataa-ainnnnreeeeeeee interim-reseate reserrr-rrtearr--reror-o---
out of superossitpo Accessed: 2024-09-27.
[56] Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002. 2020.
[57] Mattco Spinelli. Advanced style transfer with the mad scientist node. YouTube video, 2024. URL
https://www. youtube
Accessed: 2024-09-17.
[58] Adly Templeton and Tom Conerly al. Scaling monosemanticity: Extracting interpretable features from claude sonnet, 2024. URL |attps:// http://rantttrmee-ecccrroroeoeop circuitss.ub/2222.
scallugg-monoemmaticccry Accessed: 2024-09-27.
[59] Andrey Voynow, Qinghao Chu, Daniel Cohen Or, and Kfir Aberman. p+: Extended textual conditioning text-to-image generation. arXiv preprint (XXii:2333.0922. 2023.
[60] Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine, edi- tors, Advances in Neural Information Processing Systems, volume 36, pages -33349, Curran As- sociates, Inc., 2023. URL https::: http:ps:///pooceeingg..ooo www.::/s...niis..ccc.ccccoo https//aapppp.e.ieee/eppeeeeeee Ff22216660000000caeeeee822---- Conferenee.ccoecceceeeaaao pdf
[61] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari $ Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on muchine learning, pages 23965-23998. PMLR, 2022.
[62] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint 153499900..1549, 2021.
[63] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018
[64] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[65] Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong- Yee Lee, Oliver Deussen, and Changsheng Xu. Prospect: Prompt spectrum for attribute-awar aware personalization of diffusson models. ACM Transactions on Graphics (TOG), 42(6):1 2023.
14

---

A SAEPaint: Our Feature Editor Application
We host SAE-based feature editing app for you try adding and subtracting SAE features during a forward pass. To find interesting features to try, you can read the rest of the supplementary material or you can try some the ones of the ones we list on the app landing page. Altematively, what works best if you want to explore the features yourself is to generate images for which you believe your features of interest should be on and have look their activation masks the "Generate" tab, For
SDXL Turbo App: SAES
SDXL Base App:
Flux App: https: httpgs//nnngggiuigaeooeereeeeeer
Notice that while we trained on SDXL Turbo 1-step mode the same features without additional training also work for 4 steps and even for the base model with, e.g., 25 steps.
B SDXL Base
SDXL's default setting leverages classifier-free guidance to condition the generated image on a text prompt. We found that in this setting turning SAE features works best, when adding them the text-conditioned forward pass and subtracting them from the unconditional forward pass, see Fig.
(a) reference
(b) add to both
(c) add to cond.
(d) ours
Figure When using SDXL with classifier-free guidance and adding the #4977 "tiger texture" feature naively during both the conditional and unconditional forward pass its effects inhibit each other, see (b). In (b) we used twice the intervention strength as in (c) and (d), yet the tiger texture corresponding to feature #4977 is not visible. In (c) add it only to the conditional forward pass, which works. In (d) we add the feature to the conditional forward pass and subtract it from the unconditional one, which we found works best.
C Flux
Training settings We train a SAE on layer 18 activations of Flux-schnell 1-step. We choose layer 18 because we empirically find that its activations have higher norms than other layers. Additionally, all other exploratory experiments that we tried on FLUX, e.g., ablating layers, patching activations, simple activation steering, all consistently showed that layer 18 is high impact layer. To train the SAE, we sample million prompts from LAION-5B (53) and input them to Flux-schnell, then we randomly sample 10% of the activations (output- input) in the image stream of layer 18 (so for each prompt we get [64 x 64 x 0.1] 3072-dimensional vectors).
The trained SAE has expansion factor of 4 (thus its hidden dimension is 12288) and k 20. All other hyperparameters and the training loss are detailed in K
Features injection Features learned on Flux-schnell 1-step can be used on Flux-schnell 4-steps as well as Flux-dev (we report examples with steps). To inject feature in new generation, we simply rescale it by strength factor and add it to the output of every layer starting from layer 18, finding that this way we can achieve high-quality results. Figures and show some examples of feature injections with varying strengths.
15

---

#330: weight
#2000: wiigt 40
#336L: wiight 50
#338L weight
#3301: weight 20
#2362 wiight dO
#11000 weigh
#13300
#11000
#11080 wight 60
#33898
#11800
neigt
#2141 40
weight 50
M weight
#IPALL wegge
weight DO
#13867 wight
#22012 weigh
#1191h weigh
#23807 wight 60
#32017 wHight 70
#SUMT wwight
#7213
60
Figure Feature injections on Flux-schnell 4-steps generations.
D Finding Causally Influential Transformer Blocks
We narrow down design space of the 11 cross-attention transformer blocks (see Fig. T0 to those with the highest causal impact the output. In order to assess their causal impact the output we qualitatively study the effect of individually ablating each of them (see Fig. TT. can be seen in Fig. M each of the middle blocks down. 2. 1, mid. O, up 0. O, up. 0. 1 have relatively high impact on the output respectively. In particular, the blocks down. 2. 1 and up. O. stand out. It seems like most colors and textures are added in up. 0. which in the community is already known as "style" block [57]. Ablating down. 2. which also already known in the community as "composition" block, impacts the entire image composition, including object sizes, orientations and framing. The effects of ablating other blocks such as mid. and up. 0. are more subtle. For mid. 0 difficult to describe in words and up. 0. seems add local details to the image while leaving the overall composition mostly intact.
We also provide a quantitative version of this experiment in Tab. 1 As can be seen some of the resnet blocks also exhibit similarly strong effects. We added this experiment during our NeurIPS rebuttal and think it is promising future direction to investigate these blocks well.
16

---

#330: weight
#3000: miight 40
#336:: wiiht
#330L wight
#300 weight 70
#226) weight
#21880: wight.0
#13800
#11000
#21080: wigh
#3388
#11800
meihh
#3741: migg1-0
wiight
#3741 weggt
weight
#13907: wright
#12817 WROTE
#1091T weight 30
#13807 wight 60
#12017 wHich
#SUMA wright
#7t1h
#7713
#721
#771
#3233
Figure 9: Feature injections on Flux-dev 25-steps generations.
E
Interventions in the Multi-Step Setting
In addition to our quantitative analysis from the main paper showing that features stabilize fast and are relatively shared across timesteps, here, we performed series of experiments to also qualitatively assess the impact of performing interventions across multiple timesteps and also on subsets of timesteps. See Fig.
and
Broadly, these results are aligned with what one would expect. Intervening from the beginning the end leads big perturbations of the original generation. Starting the interventions later denoising steps keeps more of the original generated image intact. Interestingly, the sliding window of interventions shows that the different transformer blocks can have different effective ranges, e.g.. up. 0. 1 features start working later than down. 2. features.
RIEBench: Representation-base Image Editing Benchmark
For each our PIEBench adaptation's edit categories, we implement corresponding feature transport interventions. When selecting feature indices in SDXI. Turbo, aggregate them across spatial positions and timesteps by taking the mean across these dimensions. In FLUX Schnell, we only considered the one-step setting and thus only have to aggregate over spatial locations. The different
17

---

DOWN.O (no attn)
UP.2 (no attn)
UP.1.2
DOWN.1.0
UP.1.1
DOWN. 1.1
UP.1.0
UP.0.2
DOWN.2.0
UP.0.1
DOWN.2.1
UP.0.0
ResNet Upsampler
MID.0
Downsampler
Figure 10: Cross-attention transformer blocks in SDXL's U-net.
baseline
down.1.0
down. 1.1
down 2.0
down.2.1
mid.0
up 0.0
up.0.1
up 0.2
up 1.0
up.1.1
up
Figure 11: We generate images for the prompts "A dog playing with a ball cartoon.", "A photo of a colorful model.", "An astronaut riding on pig on the moon.", "A photograph of the inside of subway train. There are frogs sitting the seats. One of them is reading newspaper. The window shows the river the background." " and "A cinematic shot of a professor sloth wearing a tuxedo at BBQ party." while ablating the updates performed by different cross-attention layers (indicated by the titles). The title "baseline" corresponds to the generation without interventions.
18

---

Table Causal impact of individual resnet and attention blocks. Each block ablated indepen- dently, and the resulting image is compared to the unperturbed output using the LPIPS distance. We report the mean over 20 randomly generated prompts. Higher values indicate greater causal influence.
Block Type
Block Name
LPIPS Score
ResNet
down blocks. 0.rmmets..
0.733
ResNet
down Glocks...resmeee.
0.415
Attention
down_blocks. Iaattccin.
0.185
Attention
0.161
ResNet
Nown__blocks.I.resnntt.]]
0.457
ResNet
Nown_bllock.I.resneets..
0.306
Attention
down_bblokss2.attemtiinn..
0.194
Attention
down_bllcks..2atteeniiort
0.512
ResNet
down_bblocks.2.resnnttt
0.523
ResNet
down blocks...resnetss.
0.270
Attention
mid block.attentions
0.345
ResNet
mid block.resmets.0
0.240
ResNet
mid blocckresnets..
0.132
Attention
up up up
0.395
Attention
0.521
Attention
0.281
ResNet
up www_blockk..reesenee
0.348
ResNet
up blw.cks......enesss........ up blocks.0.resnett.. up 0latess...ttenntions..0
0.170
ResNet
0.157
Attention
0.252
Attention
up blo.bblo......o Lattems..ttttmtioos...
0.217
Attention
up Jattessn.
0.254
ResNet
up up_blocks.11 w...bokk.....snets.......
0.199
ResNet
up_bloccs.
0.168
ResNet
up up_blocks. 1.rsset..
0.201
ResNet
up up_blockss..resneess
0.275
ResNet
up ap_blocks.2.resnett
0.703
ResNet
up_blocks.2.resnets..
0.851
interventions for the different categories mainly differ in where the features are collected and whether they are inserted using the target mask or the source mask. We don't 't preserve spatial information when adding and subtracting feature coefficients neuron activations block activations, i.e., at each location within the respective mask the same update is performed. We refer to the mask computed on the target forward pass as target mask and the one computed on the source forward pass as source mask. We split the interventions into three different types:
1. Change interventions:
We add the top features with coefficients obtained from the source forward pass using also the source mask and we subtract the bottom features with coefficients from the target forward pass using the target mask. Change object (SDXL Turbo Fig. 26 and
Fig.
13t
FLUX Fig. 35). content (SDXL Turbo Fig.
29
FLUX Fig.
38
pose (SDXL Turbo
Fig.
30
FLUX Fig.
color (SDXL Turbo Fig. FLUX Fig.
40p.
material (SDXL Turbo
Fig.
FLUX Fig.
background (SDXL Turbo Fig.
33
FLUX Fig 42
style (SDXL
Turbo
Fig.
34
FLUX Fig. 43
fall within this category.
2. Add object:
FLUX Fig. 36 requires spe- cial treatment. Here we use the source mask both forward passes both to select and subsequently add the top and subtract the bottom features.
The add object intervention (SDXL Turbo Fig. 27
3. Delete object:
The delete object intervention (SDXL Turbo Fig. 28 FLUX Fig. 37 also requires special treatment. Here we use the target mask in both forward passes both to select and subsequently add the top and subtract the bottom features.
Neuron selection. Let H( E R denote a neuron activation map for neuron p at layer & and timestep t. First, to make layers comparable each other normalize by L2 norm H [6]e,
19

---

Festure: up Pirate strength-3] Cat strengtt-3
10.35
$3.25
10
19.03
20235
Rafannce
05
$30
10-5
1529
20:25
Figure 12: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "tiger texture feature". We intervened with this feature across the entire face of the pirate and across the entire cat except its ears. These results are from our first working SAE's with k = 10 and n2 = 5120. This is a preview, please find the remaining multi-step intervention figures at the bottom of the document after the text.
H[d [MPll Then, similar as in our our feature selection step, we aggregate the neuron activations
by taking the mean over timesteps, i.e., [[] = + ΣT [[[ To determine which neurons to score >/4, that quantifies the absolute difference
transport, rank each neuron p according in the magnitude of across the source and target, aggregated over the respective grounded SAM2 masks and normalized by the sum of all feature magnitudes:
M
M
14.
E
where h[pppp Jarsre EnJE A1 [1] and A[0) parier EnjEma H[ In equation we use
1
absolute difference because GEGLU [56] neurons take positive and negative values (in contrast to SAE feature coefficients that are always positive). Again, to select neurons among multiple layers simply concatenate the layer specific score vectors > 12 i.e., 7 >/4]- ..lllel-eeeeeee [Lau]. --]]aa)) where denotes concatenation, before ranking. For neurons n = 5, 120 for a single layer and there are 40 layers in total within our considered transformer blocks, resulting in 7 E 20000000
FLUX. As mentioned in App. C for FL.UX we found that interventions are more effective when performing the same update across multiple layers. Thus, the FLUX figures Fig. 35 Fig. always have multi-layer interventions (top rows with y-label "multi") and single-layer interventions (bottom rows with y-label "single"). The multi-layer interventions are performed from layer 18 to 56, i.e., for 39 layers. They are effective using small intervention strengths. In contrast, the single-layer interventions FLUX require big intervention strengths to have a causal influence on the output. While we need to further investigate this our best guess is that multi-layer interventions interact more gracefully with normalization layers than single layer ones and thus introduce less artifacts while achieving a similar effect to using very high intervention strength at single layer. Further, we omit neuron interventions from the qualitative examples because they did not work in FLUX when performed only using neurons from layer 18.
i we
20

---

56. S42
S40 54E
Figure 13: Example for edit category 1: "change object". Original prompt (target): "a cute little bunny with big eyes", edit prompt (source): "a cute little pig with big eyes". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used collect the features are shown but in this example they would select the entire foreground objects respectively. This is preview, please find the figures for the remaining edit categories the bottom of this document after the text.
Fig. 14 shows quantitative evaluation of our FLUX SAE interventions. We omitted neurons form this plot because our neuron interventions based on the neurons from layer 18 did not significantly increase CLIP similarity with the edit prompt in any of the categories. As can be seen, the multi- multi-layer interventions outperform the single-layer ones both for steering as well as for SAE interventions on change object, add object, change content, color, material, background, style tasks. On the delete object and change pose tasks none of the considered methods works well. The change pose tasks is hard achieve using our current RIEBench setup that in essence consists of interventions adding an update vector to a masked area. We think the bad performance on the delete object task can be overcome by improving the delete interventions. E.g., by improving the feature selection strategy or by subtracting the relevant features everywhere in the image and not just locally.
F1 Feature Visualization Techniques
We introduce our methods used for feature visualization used in Fig. 15 Informally, given feature, spatial activations (denoted by hmap) highlight the regions of image where the feature activates during the generation process. Activation modulation (A. columns) refers to the intervention process in which the feature activations are enhanced or diminished. This technique is used to demonstrate how the manipulation a feature's 's value affects the generated image. Finally, empty-prompt interventions (B. column) illustrate the isolated role of the feature by disabling all other features during generation conditioned on an empty prompt. In the remainder this section, we provide formal definitions and details.
Spatial activations. We visualize sparse feature map S R containing activations feature p across the spatial locations by up-scaling it to the size of the generated images and overlaying it as
21

---

SAE [18]
Steer [18, 56]
Steer [18]
20
20
50
5-10
1=10
5-20
sot
1-20
Change object
Add object
Delete object
X
better
CLP increase
0.03
0.03
0.03
0.02
0.02
0.02
0.00
0.00
0.00
Change content
Change pose
Change color
CLIP Increase
0.03
0.03
0.03
0.02
0.02
0.02
0.00
0.00
0.00
0.0
0.6
0.0
0.6
0.0
Change material
Change background
Change style
0.04
0.04
0.04
*
CUP Increase
0.02
0.02
0.02
0.00
0.00
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.5
0.0
0.2
0.6
LPIPS
LPIPS
LP
Figure 14: FLUX Schnell one-step evaluation on RIEBench; LPIPS with original image (x-axis) versus increase in CLIP similarity with the edit prompt (y-axis) from feature-transfer interven- tions using SAF features, and activations across nine edit categories from PIEBench [26]. We omitted neuron experiments because they failed to increase CLIP similarity with edit prompt. Features from the edit prompt (within SAM2 segmentation mask) are transferred into the original prompt's forward pass. We experiment with adding SAE/steering based updates a single layer (SAE red, stecring yellow), which is the standard way to do this, as well as adding the same feature on multiple layers (SAE blue, steering green). Strengths (5= for single-layer and s=0.7.s=1.0.s=1.. for multi-layer) and numbers of features transported (1, 50) are indicated in the legend. We omit all settings that fail to increase CLIP similarity.
a heatmap over the generated images. In the heatmap, red indicates the highest feature activation, and blue represents the lowest non-zero one.
Top dataset examples. For a given feature p, we sort dataset examples according to their average spatial activation
ap
S R.
(10)
wh
1=1
We equation 10 to define the top dataset examples and sample from the top 5% quantile of the activating examples (a, > 0). We will refer to them as top 5% images for a feature p.
Note that S always depends an embedding of the input prompt and input noise 21, via Siz(c, ENC( (Nj(AD; 21)), which we usually omit for ease of notation. As a result, a also depends on and 21. When we refer to the top dataset examples, we mean our (c, pairs with the largest values for (c, 21).
22

---

Activation modulation. We design interventions that allow us to modulate the strength of the pth feature. Specifically, we achieve this by adding or subtracting a multiple of the feature p on all of the spatial locations j, proportional to its original activation Si
DD'j ADj + F.
(11)
in which A Diy is the update performed by the transformer block before and AD' after the interven- tion, 3 E R is modulation factor, and f, is the pth learned feature vector. In the following, we will refer to this intervention as activation modulation intervention.
sj
Note that Süj can be also freely defined allowing for the application of sparse features to arbitrary images and spatial positions (refer to Fig. D for examples).
Activation on empty context. Another way of visualizing the causal effect of features is to activate them while doing forward pass on the empty prompt c(""). To do so, we turn off all other features at the transformer block é of intervention and turn on the target feature p. Formally, we modify the forward pass by setting
Doge' Dij + rkppEp
(12)
in which Dogdl" replaces residual stream plus transformer block update, D is the input to the block, f is the pth learned feature vector, 7 E IR is hyperparameter to adjust the intervention strength, and UP is a feature-dependent multiplier obtained by taking the average activation across positive activations of p (collected over a subset of 50.000 dataset examples). Multiplying it by & aims to recover the coefficients lost by setting the other features to zero. Further in the text, will refer to this intervention as popptpppompt intervention, and the images generated using this method with y set to 1, as empty-prompl intervention images.
Note that we directly added/subtracted feature vectors the dense vectors for both intervention types instead of encoding, manipulating sparse features, and decoding. This approach helps mitigate side effects caused due to reconstruction loss (see App.
G Case Study: Most Active Features on a Prompt
we depict the features with the highest average activation when processing the prompt: "A cinematic shot of professor sloth wearing a tuxedo at BBQ party". We discuss the transformer blocks in order of decreasing interpretabillity
Combining all our feature visualization techniques, in Fig. 13
Down.2.1 seems to contribute towards the image composition. Several features seem to relate directly to phrases of the prompt: 4539 "professor sloth", 4751, 1226. "wearing a tuxedo", 2881, 567, 3119, 2345 "party".
Turning off features (A. -6.0 column) removes elements and changes elements in the scene in ways that align with heatmap (hmap column) and the top examples (C columns): 1674 removes the light chains in the back, 4608 the umbrellas/tents, 4539 the 3D animation-li like sloth face, 567 people in the background, 31,9, 2345 some of the light chains, and, 4751 changes the type of suit, 1226 the shirt. Similarly, enhancing the same features (A. 6.0 column) enhances the corresponding elements and sometimes changes them.
Activating the features on the empty prompt often creates related elements. Note that, for the fixed random seed we use, the empty prompt itself looks like a painting of a piece of nature with lot of green and brown. Therefore, while the prompt is empty the features active during the forward pass are not and due to the layers that we don't 't intervene on still contribute to the images.
While top dataset examples (C.0, C.I columns) and also mostly agree with the feature activation heatmaps (hmap column), some of them add additional insight, e.g., 2881, which activates on the suit, seems to correspond to (masqueraded) characters in a (festive) scene, 3119 seems to be about party decorations in general and not just light chains, 2345 seems to react to other celebration backgrounds as well.
intervention (B. column)
Up.0.1 transformer block indeed seems to contribute substantially to the style of the image. They are hard to relate directly to phrases in the prompt, yet indirectly they do relate, E.g. the illumination (2727) and shadow (500, 1700) effects probably have something to do with "a cinematic shot" and
23

---

1674
46on
(a) Top 9 features of down. 2.1
(b) Top 9 features 0.
hmap
-8.0
8.0
C
C
2.2
hmap
-6.00
16.0
C.
1.
4755
4735
2013
B108
Sc61
306 2372
(c) Top 9 features of 000
(d) Top 9 features of mid.0
Figure 15: The top 9 features of down. 2. (a), up. 0. (b), up. O. 0 (c) and mid. 0 (d) for the prompt: "A cinematic shot a professor sloth wearing a tuxedo at BBQ party." Each row represents a feature. The first column depicts feature heatmap (highest activation red and lowest nonzero one blue). The column titles containing "A" show feature modulation interventions, the ones containing "B" the intervention of turning the feature the empty prompt, and the ones containing "C" depict top dataset examples. Floating point values the title denote / and ry values. These results are from our first working SAE's with k 10 and n, 5120.
the animal hair texture (2314) with "sloth". Beyond that several features seem to mainly contribute to the glowing lights in the background (1295, 4238, 2341).

---

Interestingly, turning on the up. O. 1 features on the entire empty prompt (B. column) results in texture-like images. In contrast, when activating them locally (A. columns) their contribution the output is highly localized and keeps most of the remaining image largely unchanged. For the up.0. O. we find it remarkable that often the ablation and amplification are counterparts: 500 (light, shadow), 2727 (shadow, light), 3936 (blue, orange), 2314 (less grey hair, more brown hair).
Up.0.0. First, we observe that up.0.0 features act very locally and we think that often requires relevant other features from the previous and subsequent transformer blocks effectively influence the image. For the empty prompt, activating these features results in abstract looking images, which are hard to relate to the other columns. Thus, we excluded this visualization technique and instead added one more example.
Most top dataset examples and their activations (C columns) are highly interpretable: 3603 party decoration, 5005 upper part of tent, 775 buttons suit, 153 lower animal jaw, 1550 collars, 2648 pavilions, 1604 right part of the image, 564 bootie. Many of the features have a expected causal effect on the generation when ablating/enhancing (B. columns): 3603, 5005, 775, 153, 1550, 564, but not all: 2221, 2648, 1604. To sum up, this transformer block seems to mostly add local details to the generation and when interventions are performed locally they are effective.
Mid.0. Again to the best of our knowledge, mid. O's role is also not well understood. We find harder to interpret because most interventions on the mid. 0 have very subtle effects. Similar to up. 0. O, we did not include the results of empty-promph interventions.
While effects of interventions are subtle, dataset examples (C. columns) and heatmap (hmap column) all mostly agree with each other and are specific enough to be interpretable: 4755 bottom right part of faces, 4235 left part of (animal) faces, 1388 people in the background, 1935 is active on chests, 473 mostly active on the image border, 2322 again seems to have to do with backgrounds that also contain people, 3067 active on the neck or neck accessories, and, 5102 outlines the left border of the main object in the scene. The feature 3018 is difficult to interpret.
Our observations indicate that mid. 0's features encode more abstract concepts. Particularly, some of them are activated at specific spatial locations within images and other features potentially signify how image objects relate to each other.
H Case Study: Random Features
In this case study, we explore the learned features independently of any specific prompt. We moved the many and large figures corresponding to this section to the end of the supplementary material. In Fig. 4 and Fig. 45 we demonstrate the first and last 5 learned features for each transformer block (since SAES were initialized randomly before training, we can treat these features as a random sample). As SAES are randomly initialized before the training process, these sets can be considered as random samples of features. Each feature visualization consists of 3 images of top 5% images for this feature, and their perturbations with activation modulation interventions. For down. 2. 1 and up O. we also include the empty-prompt intervention images. Additionally, we provide visualizations of several selected features in App. H
Fig. 46 and demonstrate the effects of their forced activation on unrelated prompts in App. H Fig.
Feature plots. We provide the same plots as in Fig. 44 but for the last six feature indices of each transformer block in Fig. 45 and the corresponding prompts in Table 7 Additionally, provide some selected features for down. 1 and 0. in Fig. 46 and the corresponding prompts in Table g
Intervention plots. Additionally, we provide plots in which we turn on features from Fig. 46 but in unrelated prompts (as opposed to top dataset example prompts that already activate the features by themselves). For simplicity here we simply turn on the features across all spatial locations, which does not seem be a well suitable strategy for up. 0. 1, which usually acts locally. To showcase, the difference we created one example image in Fig. 16 in which we manually draw localized masks to turn on the corresponding features.
SSDXL Turbo does not utilize positional encodings for the spatial locations in the feature maps. Therefore, we did a brief sanity check and trained linear probes to detect i. given D These probes achieved high accuracy holdout set: 97. 97.9%, 98. 48%, 99. 44,, 95.57% for down. 2. 1, mid. O, up. 0.0, up. 0.1.
25

---

Active interventions
Orig.
240
4955
240
4977
240
3718
240
06
(a) Intervention history
(b) Result
Figure 16: Local edits showcase 0..0' ''s ability to locally change textures in the image without affecting the remaining image. Multiple consecutive interventions are possible (a). The first in (a) row depicts the original image and each subsequent row add intervention by drawing heatmap with a brush tool and then turning on the feature labelling the row only on that area. The other number (240) is the absolute feature strength of the edit. Figure (b) shows the final result in full resolution (512x512). These results are from our first working SAE's with k 10 and ny 5120.
I Quantitative Evaluation of the Roles of the Blocks
In this section, we follow up qualitative insights by collecting quantitative evidence.
L1 Annotation Pipeline
Feature annotation with an LLM followed by further evaluation is common way assess feature properties such as specificity, sensitivity, and causality 7 We found applicable to the features learned by the down. 2. transformer block, which have strong effect on the generation. Thus, they are amendable to automatic annotation using visual language models (VLMs) such GPT-4o [38] In contrast, for the features other blocks with more subtle effects, captions to be unsatisfactory. In order to caption the features of down 2. 1, we prompt GPT-4o with sequence of 14 images. The first five images are irrelevant to the feature (i.e., the feature was inactive during the generation of the images), followed by progression of 4 images with increasing average activation values, and finished by five images with the highest average activation values. The last nine images are provided alongside their so-called "coldmaps": version of image with weakly
found VLM VLM -LM-generated
26

---

active and inactive regions being faded and concealed. The prompt template and examples of the captions can be found in the App. g
1.2 Experimental Details
We perform a series of experiments to get statistical insights into the features. We report the majority of the experimental scores in the format (S). When the score is reported in the context of a SDXL. Turbo transformer block, means that we computed the score for each feature of the block and set M and S to mean and standard deviation across the feature scores. Note that S does not represent the error margin of M, as the actual error margin is much lower Therefore, almost all the differences in the reported means are statistically significant. For the baselines, we calculate the mean and standard deviation across the scores of a 100-element sample.
Table 2: Specificity, texture score, and color activation for different blocks and baselines.
Block
Specificity
Texture
Color
Down. 2.1 SAE
0.76 (0.10)
0.16 (0.02) 86.2 (14.9)
Doun.2.1 Neurona
0.65 (0.09)
Down. 2.1 PCA sll
0.58 (0.06)
Down.2.1 PCA
0.66 (0.08)
Doun.2.1 PCA 100
0.64 (0.08)
Do. PCA 500
0.61 (0.07)
0.70 (0.10)
0.14
84.7 (16.3)
Kid Neurons
0.67 (0.07)
Up.0.0 SAE
0.74 (0. 10)
0.18 (0.03)
86.3 (16.5)
Up.0.0 Beurone
0.67 (0.07)
Up.0.1 SAE
0.73 (0.09)
0.20 (0.02)
73.8 (20.6)
Up.0.1 Neurons
0.66 (0.08)
Randon
0.57 (0.09)
0.13 (0.02)
90.7 (54.9)
Sane Prompt
0.89
Textures
0.18 (0.02)
Interpretability. Features are usually considered interpretable if they are sufficiently specific, i.e., images exhibiting the feature share some commonality. In order to measure this property, we compute the similarity between images on which the feature is active. High similarity between these images proxy for high specificity. For each feature, we collect 10 random images among top 5% images for this feature and calculate their average pairwise CLIP similarity semantically similar the contexts are in which the feature is most active. We display the results in the first column of Table _ which shows that the CLIP similarity between images with the feature active is significantly higher then the random baseline (CLIP similarity between random images) for all transformer blocks. This suggests that the generated images share similarities when a feature is active.
2. This value reflects how
For down. 2. 1 we compute an additional interpretabiliiyy score by comparing how well the generated annotations align with the top 5% images. The resulting CLIP similarity score is 0.21 (0.03) and significantly higher then the random baseline (average CLIP similarity with random images) 0.12 (0.02). To obtain an upper bound on this score we also compute the CLIP similarity to image generated from the feature annotation, which is 0.25 (0.03).
Causality, We can use the feature annotations to measure a feature's 's causal strength by comparing the empty prompt intervention images with the caption The CLIP similarity between intervention images and feature caption is 0.19 (0.04) and almost matches the annotation-base interpretability score of 0.21 (0.03). This suggests that feature annotations effectively describe the corresponding ampty-prompt intervention images. Notably, the annotation pipeline did not use umpty-promet intervention images to generate captions. This fact speaks for the high causal strength of the features leamed on down. 2.
Sensitivity. A feature is considered sensitive when activated in its relevant context. As a proxy for the context, we have chosen the feature annotations obtained with the auto-annotation pipeline. For each learned feature, we collected the 100 prompts from 1.5M sample of LAION-COCO with
19 Givee that M is computed over sample of 1280 elements, the confidence interval M can be estimated as M + S. 0.055. We require feature captions for the causality and sensitivity analyses, we only have them for down.
27

---

the highest sentence similarity based on sentence transformer embeddings of all al1-MiniLM-L6-v2
Next, we run SDXL Turbo on these prompts and count the proportion of generated images in which the feature is active on more than 0%, 10%, 30% of the image area, resulting in 0.60 (0.32), 0.40 (0.34), 0.27 (0.30) respectively, which is much higher than the random baseline, which is at 0.06 (0.09), 0.003 (0.006). 0.001 (0.003). However, the average scores are < and thus not perfect. This may be caused by incorrect or imprecise annotations for subtle features and, therefore, hard to annotate with VLM and SDXL Turbo failing comply with some prompts.
48.
Relatedness to texture. In Fig. 15 the empty prompt interventions of the up.0.1 features resulted in texture-like like pictures. To quantity whether this consistently happens, we design simple texture score by computing CLIP similarity between an image and the word "texture". Using this score, we compare empty-prompt interventions of the different transformer blocks with each other and real- world texture images. The results are in the second column of Table 2 and suggest that empty-prompt intervention images of up. 0. 1 and up. O.0 resemble textures and some of the down 2. images look like textures as well. For up.0. O.O, we did not observe any connection of these images to the top activating images. Interestingly, the score of up. O. 1 is higher than the one of the real-world textures dataset (Cimpoi et al. (10).
Color sensitivity. In our qualitative analysis, we suggested that the features learned on up.0. op.0.1 0.1 relate to texture and color. If this holds, the image regions that activate a feature should not differ significantly in color on average. To test that, we calculate the "average" color for each feature: this a weighted average of pixel colors with the feature activation values as weights. To determine the average color of each feature we compute it over a sample of 10 images of the feature's 's top 5% images. Then, we calculate Manhattan distances between the colors of the pixels and the "average" color on the same images (the highest possible distance is 3- 255 = 765). Finally, we take weighted average of the Manhattan distances using the same weights. We report these distances for different transformer blocks and for the images generated random prompts from LAION-COCO, We present the results in the third column of Table The average distance for the up. 0. 1 transformer block is, in fact, the lowest.
Table 3: Manhattan distances between original and intervened images at varying intervention strengths outside/inside of the feature's activation map.
Block
-10
-5
Dove. 2.1
148.2/ 116.0
124.2 94.4
101.4/ 78.7
128.9/ 105.60
Mid
/8.5
33.2/15.2
59.9/29.82 68.6/21.9
Up.0.0
105.3/ 38.4
77.7/23.7
63.6/23.3
Up.0.1
125.0 26.8
7
94.9/34.74
Intervention locality. We suggested that features learned on up. O.O and up. O. 1 primarily influence local regions of the generation, with minimal effect outside the active areas. To test this, we measure changes in the top 5% images inside and outside the active regions while performing activation modulation interventions. To exclude weak activation regions from consideration, pixel is considered inside the active area if the corresponding patch has an activation value larger than 50% of the image patches, and it is outside the active area if the corresponding patch has zero activation. Table 3 reports Manhattan distances between the original images and the intervened images outside and inside the active areas for activation modulation intervention strengths -10, -5, 5, 10. The features for up.0 O and up. O. 1 have a stronger effect inside the active area than outside, unlike down. 2. where the difference is smaller.
J Annotation Pipeline Details
We used GPT-4o to caption learned features on down. 2. 1. For each feature, the model was shown a series of 5 unrelated images, progression of images, the i-th of those corresponds to ~ i 10% average activation value of the maximum. Finally, we show 5 images corresponding to the highest average activations. Since some features are active particular parts of images, the last 9 images provided alongside their so-called "coldmaps": a version of an image with weakly active and inactive regions being faded and concealed.
The images were generated by 1-step SDXL. Turbo diffusion process on 50'000 random prompts of LAION-COCO dataset.
28

---

J.1 Textual Prompt Template
Here the prompt template for the VLM.
System. You are an experienced mechanistic interpretability researcher that is labeling features from the hidden representations of an image generation model. User. You will be shown series of images generated by machine learning model. These images were selected because they trigger a specific feature of a sparse auto-encoder, trained to detect hidden activations within the model. This feature can be associated with a particular object, pattern, concept, or a place on an image. The process will unfold in three stages:
1. *#Reerence Images: ** First, you'l IIl see several images *unrelated* the feature. These will serve as a reference for comparison.
2. Freature-Activtting Images:*# Next, you'll view images that activate the feature with varying strengths. Each of these images will be shown alongside a version where non-activated regions are masked out, highlighting the areas linked the feature.
3. **Sronngest Activators:**
Finally, you'll II be presented with the images that most strongly activate this feature, again with corresponding masked versions to emphasize the activated regions.
Your task is to carefully examine all the images and identify the thing or concept represented by the feature. Here's 's how to provide your response:
**Reasoning:** Between '<thinking>' ' and "</thinking> tags, write up to 400 words explaining your reasoning. Describe the visual patterns, objects, or concepts that seem be consistently present in the feature-activatingg images but not in the reference images.
** Expression: ** Afterward, between "<answer>' and "</answer> tags, write a concise phrase (no more than 15 words) that best captures the common thing concept across the majority of feature-activating images.
Note that not all feature-activating images may perfectly align with the concept you're 'rer describing, but the images with stronger activations should give you the clearest clues. Also pay attention to the masked versions, as they highlight the regions most relevant the feature.
User. These images are not related to the feature: [Reference Images]
User. This is a row of images, each illustrating increasing levels of feature activation. From left to right, each image shows progressively higher activation, starting with the image on the far left where the feature is activated at 10% relative to the image that activates it the most, all the way to the far right, where the feature activates at 90% relative to the image that activates the most. This gradual transition highlights the feature's growing importance across the series. Acctiaie- Images]
User. This row consists of 9 masked versions of the original images. Each masked image corresponds to the respective image in the activation row. Areas where the feature is not activated are completely concealed by a white mask, while regions with activation remain visible.) (Feature-Activating Images Coldmaps) User. These images activate the feature most strongly. (Strongest Activators) User. These masked images highlight the activated regions of the images that activate the feature most strongly. The masked images correspond to the images above. The unmasked regions are the ones that activate the feature. (Strongest Activators Coldmaps]
J.2 Example of Prompt Images
The images used to annotate feature 0 are shown in Fig. 17
J.3 Examples of Generated Captions
We present the captions generated by GPT-4o for the first and last 10 features in Table +
29

---

Table down 2. first 10 and last 10 feature captions.
Block
Feature
Caption
down.2.1
0
Orgunizationul/ttrage items documents and office supplies
Laxury kitchen interiors and designs
2
Architectural Landmarks Monumental Buildings
3
Upper body clothing and attire Natural
or Surfaces
5
Intricutelyy designed and ornamental brooches
6
Technical diugrams and instructional content
7
Feature predominantly activated visual representations of dresses Home decor textiles focusing on cushions and pillows
8
9
Eyewear: glasses sunglasses
S110
Concept of containment or organized enclosure
Groups of people
settings
5112
Modem minimalist interior design
5113
Indoor plants and greenery
5114
Feature sensitivity focused sneakers
5115
Handling or manipalating various objects
Athletic
particularly zippered sporty jackets Spectator Seating in Sporting Venues
S117
Textiles clothing materials, focus textures and
5119
Yam and Knitting Textiles
LIIFTING MASK
Domggment
Dompgnent
The images used by GPT-4o to generate captions for feature 0. From top to bottom: irrelevant images to feature 0; image progression from left right, showing increasing activation of SAE feature 0, with low activation on the left and high activation on the right; "Coldmaps" representing the image progression; images corresponding the highest activation of feature 0: "Coldmaps" corresponding to these highest activation images.
Figure 17:
30

---

K Sparse Autoencoders and Superposition
This is an extended version of Sparse Autoencoders subsection of background section.
Let h(x) E Rd be some intermediate result of a forward pass of a neural network on the input x. In a fully connected neural network, the components h(z) could correspond to neurons. In transformers, which are residual neural networks with attention and fully connected layers, h(x) usually either refers to the content of the residual stream after some layer, an update to the residual stream by some layer, or the neurons within a fully connected block. In general, h(x) could refer to anything, e.g.. also keys, queries, and values. It has been shown 62 5 that in many neural networks, especially LLMs, intermediate representations can be well approximated by sparse sums of n E N learned feature vectors, i.c..
1(x))
(x)
(13)
where sp(x) are the input-dependen coefficients most of which are equal to zero and fi..... fny R is a learned dictionary of feature vectors.
Importantly, these learned features are usually highly interpretable (specific), sensitive (fire on the relevant contexts), causal (change the output in expected ways intervention) and usually do not correspond directly to individual neurons. There are also some preliminary results on the universality of these learned features, i.e., that different training runs on similar data result in the corresponding models picking up largely the same features [5]
Superposition. By associating task-relevant features with directions in Idd instead of individual components of h(x)) E IR4, it is possible to represent many more features than there are components, ie.. ny >> d. As result, in this case, the learned dictionary vectors f,....,fr .,, cannot be orthogonal to each other, which can lead to interference when too many features are on (thus the sparsity requirement). However, it would be theoretically possible have exponentially (in d) many almost orthogonal directions embedded in Rd 13
Using representations like this, the optimization process during training can trade off the benefits of being able to represent more features than there are components in h with the costs of features interfering with each other. Such representations are especially effective if the real features underlying the data do not co-occur with each other too much, that is, they are sparse. In other words, in order to represent single input ("Michael Jordan") only small subset of the features ("person". "played basketball") is required [776]
The phenomenon of neural networks that exploit representations with more features than there components (or neurons) is called superposition [17]- Superposition can explain the presence of polysemantic neurons. The neurons, in this case, are simply the wrong level of abstraction. The closest feature vector can change when varying neuron, resulting in the neuron seemingly reacting to steering semantically unrelated things.
Sparse autoencoders. In order to implement the sparse decomposition from equation 13 containing the y coefficients of the sparse sum is parameterized by single linear layer followed by an activation function, called the encoder,
the vector
ENC(h) (W bpec) + bac).
(14)
in which h E Rd is the latent that aim decompose, a (-) is activation function, RMENG RR a learnable weight matrix and bys and dast are learnable bias terms. We omitted the dependencies h h(() and (h) that are clear from context.
Similarly, the learnable features are parametrized by a single linear layer, called decoder,
h' DBC(s) DBC g + bpre.
(15)
in which WBB8c (fil ... [f,y) E Rdxny is a learnable matrix of whose columns take the role of learnable features and 6pr is learnable bias term
In the literature this input dependence is usually omitted.
BI follows from the Rd with the dot product between any two not larger than
Lemma [25] that one can find at least exp(de? /8) unit vectors
31

---

Training. The pair ENC and DEC are trained in a way that ensures that h' is a sparse sum of feature vectors. Given dataset of latents hy..... hn. both encoder and decoder are trained jointly to minimize a proxy to the loss
mmin Wel W Upmr.Beet Buet
|14 h21l2 + A||s:|lo.
(16)
where h, h(x,),, . 8, EN;(H(xr;) (when we refer components of we use sp instead), the ||n - h is reconstruction loss, ||>:|lo a regularization term ensuring the sparsity of the activations and À the corresponding trade-off term.
In practice, I|sıllo cannot be efficiently optimized directly, which is why it is usually replaced with ||s:|| I or other proxy objectives.
Technical details. In our work, we make use of the top-k formulation from 19 in which ||s; S
ensured by introducing the top-k function TopK into the encoder:
(h) RopU( I W (h- bpee) + bax)).
(17)
As the name suggests, TopK returns vector that sets all components except the top i ones to zero.
In addition
19 use auxiliary loss handle dead features. During training, sparse feature p is
considered dead if &p remains zero over the last 10M training examples.
The resulting training loss is composed of two terms: the In=rreconstruction loss and the top-auxiliary L2- -e-reconstruction loss for dead feature reconstruction. For single latent h, the loss defined
L(h, h') || N'13 + aalh Naun 11
(18)
In this equation, the h'nx is the reconstruction based on the top kaux dead features. This auxiliary loss introduced to mitigate the issue of dead features. After the end of the training process, we observed none of them. Following [19] we set
32 and haux = 256, performed tied initialization of encoder and decoder, normalized decoder rows after each training step. The number of learned features n is set to 5120, which is four times the length of the input vector. The value of k is set to 10 as good trade-off between sparsity and reconstruction quality. Other training hyperparameters are batch size: 4096, optimizer: Adam with learning rate: 10-4 and betas: (0.9, 0.999).
L SAE Training Results
We trained several SAEs with different sparsity levels and sparse layer sizes and observed no dead features. To assess reconstruction quality, processed 100 random LAION-COCO prompts through a one-step SDXL Turbo process, replacing the additive component of the corresponding transformer block with its SAE reconstruction.
The explained variance ratio and the output effects caused by reconstruction are shown in Table 5 Fig. 18 presents random examples of reconstructions from an SAE with the following hyperparame- ters: k = 10, ny = 5120, trained on down 2. The reconstruction causes minor deviations in the images, and the fairly low LPIPS [x] and pixel distance scores also support these findings. However, to prevent these minor reconstruction errors from affecting our analysis of interventions, we decided to directly add or subtract learned directions from dense feature maps.
32

---

Figure 18: Images generated from 10 random prompts taken from the LAION-COCO dataset are shown in the first row. In the second row, down. 2. updates replaced by their SAE reconstructions (k: = 10, n = 5120). The third row visualizes the differences between the original and reconstructed images.
Table 5: Distances and explained variance ratio in generated images. "Mean" represents the average pixel Manhattan distance between original and coonssructionninterveee images, with a maximum possible value of 765. "Median" represents the median Manhattan distance per pixel, averaged over all images. 'LPIPS' refers to the average LPIPS score, measuring perceptual similarity. "Explained variance ratio" denotes the ratio of variance explained by the trained SAEs the total variance.
Configuration
Mean
(V)
down.2.1
83.29
50,04
0.3383
56.0
mid.0
$2.64
26.82
0.2032
640
pp000
55.89
30.69
0.2276
44.8
up.0.
$2.67
34.53
0.2073
50.3
down.2.1
mid.0
48.82
24.60
0,1845
50.8
$120
0.0
49.19
25,86
0.1969
57.2
0p
47.50
31.11
0,1775
59.5
down.2.1
73.63
41.79
0.2893
62.8
mid0
640
0p00.
48.43
25.80
0,1908
52.5
up.0.1
43.06
26.85
0.1638
58.7
10
down.2.1
64,97
34.77
0,2582
73.7
5120
mid.0
44.02
21.72
0.1622
58.8
pp000
21.54
0,1624
up.0.1
39.77
24.84
0,1453
67.1
59.29
31.47
0.229]
mid0
39.95
19.44
0,1459
60.0
0p0
40.15
21.06
0.1499
60.9
0.
0.1196
dowa.2.1
56.37
29.04
0,2190
78.8
mid0
5120
0p0
35.73
18.03
0,1302
70.6
pp...I
30.31
17.22
0.1
33

---

Feature: 2301 Pirat strangth-10 Cat strangtims
35:25
20-25
fafeence
530
16:25
15:00
20:15
5:23
12:25
20:23
Refowere
05
3.00
10:15
15.20
20.25
U
O
Figure 19: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "evil feature". We intervened with this feature across the entrire spatial grid. These results are from our first working SAE's with 10 and n = 5120.
34

---

Peature: c0M1 Pirae strengthws Cat strungthi55
10-25
35:25
20-25
fafeerce
530
16:55
15:00
20-15
10:15
13:25
Refowere
R.S
100
10.15
15.20
20.25
O
Figure 20: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "cartoon feature"'. We intervened with this feature across the entire spatial grid. These results are from our first working SAE's with 10 and n = 5120.
35

---

feature: 1p011941 Prate strength-s Cat strungth=
Referanc
0.2
5300
10:25
15:20
20-25
Refowee
310
13.29
20.25
Figure 21: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "ear feature". We intervened with this feature on the cars. These results are from our first working SAE's with k 10 and n. 5120.
36

---

Neature: 120 PPrate strengthe-3 Cat 3trungthw3
1025
20-25
fafeerce
530
16:25
15:00
20-15
12:25
Refower
0.5
3.00 10.15 15.29
20.25
Figure 22: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "beard feature". We intervened with this feature the chin/beard area. In the pirate subtracted this feature. These results are from our first working SAE's with k 10 and n = 5120.
37

---

Feature: Prate 3trangthm3 strengthe2
16-55
35:25
20-25
Rafeerce
530
16:25
15:00
20:15
10:15
13:25
20:25
Refowere
0.5
3.00 10.15
15.29
20.25
Figure 23: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "furry feature". We intervened with this feature across the entire beard and face of the pirate and across the entire cat except its ears. These results are from our first working SAE's with k 10 and n, = 5120.
38

---

Nature: up-4972 Pirate strungthw3 Cat strungtoo
16-55
20-25
Rafeerce
530
16:55
15:00
20-15
10:15
12:25
30:33
Refoweee
05
310
10.15
15.20
20.25
Figure 24: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "tiger texture feature". We intervened with this feature across the entire face of the pirate and across the entire cat except its ears. These results are from our first working SAE's with k 10 and n = 5120.
39

---

Nature: up15 Pirae strength=3 strungtoo
Calenncc
16:55
20-25
Rafeerce
5:00
16:25
15:00
20-25
10:15
12:25
20:25
Refowere
0.5
1.0 10.15
15.29
20.25
Figure 25: Performing interventions across different time intervals. For each prompt there are two rows, the first row contains ranges 0-25, 5-25, 10-25, 15-25, 20-25 and the second one 0-5, 5-10, 10-15, 15-20, 20-25. We would describe this feature as "giraffe pattern feature". We intervened with this feature across the entire face of the pirate and across the entire cat except its ears. These results are from our first working SAE's 's with k 10 and n = 5120.
40

---

22
Map 31
Map 322
704
Figure 26: SDXL. Turbo 4-step interventions; Example for edit category 1: "change object". Original prompt (target): "a cute little bunny with big eyes", edit prompt (source): "a cute little pig with big eyes". Source and target refers from where extract features (source) and where insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the entire foreground objects respectively.
41

---

bape 144-48 12--0
114 X31
Figure 27: SDXL. Turbo step interventions; Example for edit category 2: "add object". Original prompt (target): "a cat", edit prompt (source): "a cat with gold chain and star its head". Source and target refers from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the cat's gold chain from the source forward and also use the same area in the target forward pass.
42

---

Figure 28: SDXL. Turbo 4-step interventions; Fxample for edit category 3: "delete object". Original prompt (target): "a cat wearing headphones gray background", edit prompt (source): "a cat gray background". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used collect the features this example they would select the headphones in the target forward and the same area in the source forward pass. This example showcases frequent failure mode of our intervention where the deleted object (re Jappears in different location in the image.
shown but in
43

---

14-445
Figure 29: SDXL. Turbo 4-step interventions; Example for edit category 4: "'hange content".. Original prompt (target): "a detailed oil painting of calm beautiful woman with stars in her hair", edit prompt (source): "a detailed oil painting of laughing beautiful woman with stars her hair". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the woman's 's face in both forward passes.
44

---

12--000
12.22
May 31
2E4
Figure 30: SDXL. Turbo 4-step interventions; Example for edit category 5: "change pose". Original prompt (target): "a cartoon dog laying down on the ground", edit prompt (source): "a cartoon dog jumping up from the ground". Source and target refers from where extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but this example they would select the dogs in both forward passes.
45

---

32
332
104
Figure 31: SDXL Turbo 4-step interventions; Example for edit category 6: "change color". Original prompt (target): "a woman wearing red hat and red dress", edit prompt (source): "a woman wearing green hat and red dress". Source and target refers from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are shown but this example they would select the hats in both forward passes.
46

---

50
500
Figure 32: SDXL Turbo 4-step interventions; Example for edit category 7: ''hange material". Original prompt (target): "a drawing of young man with blue eyes", edit prompt (source); "a drawing of a young robot with blue eyes". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but this example they would select the face the man in the target and the robot source forward pass.
47

---

bape 14-442
Figure 33: SDXL. Turbo 4-step interventions; Example for edit category 8: "change background". Original prompt (target): "illustration of woman meditating in yoga posc", edit prompt (source): "illustration woman meditating in yoga pose in the sky with stars". Source and target refers to from where we extract features (source) and where insert them (target). Grounded SAM2 masks used collect the features are not shown but in this example they would select the backgrounds in both forward passes.
48

---

Figure 34: SDXI. Turbo step interventions; Example for edit category 9: "change style". Original prompt (target): "a photograph a clown with colorful hair", edit prompt (source): "a clown in pixel art style with colorful hair". Source and target refers to from where we extract features (source) and where insert them (larget). In this edit category used features from the entire spatial grid. From this example it becomes clear that for this edit category we should select fewer features and probably lower strength.
49

---

mubstt 27
urggoaa 20 20.3
Figure 35: FLUX Schnell 1-step interventions; Example for edit category 1: "change object". Original prompt (target): "a cute little bunny with big eyes", edit prompt (source): "a cute little pig with big eyes". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used collect the features would select the entire foreground objects respectively. The yllabels indicate interventions strength and whether single- multi-layer interventions are used. The column titles indicate the intervention types and numbers of features transported.
not shown but in this example they
50

---

S 23
wrpp. 2013
Figure 36: FLUX Schnell I-step interventions; Example for edit category 2: "add object". Original prompt (target): "a cat", edit prompt (source): "a cat with gold chain and star its head". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features select the cat's gold chain from the source forward and also use the same area in the target forward pass. The y-labs indicate interventions strength and whether single- multi-layer interventions used. The column titles indicate the intervention types and numbers of features transported.
not shown but in this example they would
51

---

Beggt
5A5
$
SAS
S46
Streding
Source
mal S.A.
Th PLbtaa 3.5
THAT 2A.
SHOB so 20.3
Figure 37: FLUX Schnell I-step interventions; Example for edit category 3: "delete object". Original prompt (target): "a cat wearing headphones on a gray background", edit prompt (source): "a cat on gray background". Source and target refers to from where extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the headphones in the target forward and the same in the source forward pass. This example showcases frequent failure mode of our intervention where the deleted object Japerss in different location in the image. The y-labels indicate interventions strength and whether single- or multi-layer interventions are used. The column titles indicate the intervention types and numbers of features transported.
52

---

Baget
5
Steeding
Source
1
Segeea St 253
Figure 38: FLUX Schnell 1-step interventions: Example for edit category 4; "change content". Original prompt (target): "a detailed oil painting of calm beautiful woman with edit prompt (source): "a detailed oil painting of laughing beautiful woman with stars in her hair". Source and target refers from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the woman's 's face in both forward passes. The y-labels indicate interventions strength and whether single- or multi-layer interventions are used. The column titles indicate the intervention types and numbers of features transported.
her hair".
53

---

prgpe ai 20.3
Figure 39; FLUX Schnell I-step interventions; Example for edit category "change pose". Original prompt (target): "a cartoon dog laying down on the ground", edit prompt (source): "a cartoon dog jumping up from the ground". Source and target refers to from where we extract features (source) and where insert them (target). Grounded SAM2 masks used to collect the features not shown but in this example they would select the dogs in both forward passes. The y-labels indicate interventions strength and whether single- multi-layer interventions used. The column titles indicate the intervention types and numbers of features transported.
54

---

Wab S.F
US OE
urgpoat 20 2013
Figure 40: FLUX Schnell I-step interventions; Example for edit category 6: "change color". Original prompt (larget): "a woman wearing red hat and red dress", edit prompt (source): "a woman wearing green hat and a red dress". Source and target refers to from where we extract features (source) and where insert them (target). Grounded SAM2 masks used to collect the features not shown but in this example they would select the hats in both forward passes. The y-labels indicate interventions strength and whether single- lulti-layer interventions used. The column titles indicate the intervention types and numbers of features transported.
55

---

Baget
5A6
SAE
Sa
Steeiin
Source
CT DE STAL
DE OEARU
DC SETE JPA
DAT SEAO
SPORO SL. 2013
Figure 41: FLUX Schnell 1-step interventions; Example for edit category 7: "change material". Original prompt (larget): "a drawing of young man with blue eyes", edit prompt (source): "a drawing of a young robot with blue eyes". Source and target refers to from where we extract features (source) and where we insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the face of the man in the target and the robot in the source forward pass. The yllabels indicate interventions strength and whether single- or multi-layer interventions are used. The column titles indicate the intervention types and numbers of features transported.
56

---

Baggtt
SA6
Steeding
mAba S.
anggoat 52
wAR 208 5.
argeot at 2013
Figure 42: FLUX Schnell I-step interventions; Example for edit category 8: "change background"; Original prompt (target): "illustration of woman meditating in yoga pose". edit prompt (source): "illustration of a woman meditating in yoga pose in the sky with stars".. Source and target refers to from where we extract features (source) and where insert them (target). Grounded SAM2 masks used to collect the features are not shown but in this example they would select the backgrounds in both forward passes. The y-labels indicate interventions strength and whether single- or multi-layer interventions are used. The column titles indicate the intervention types and numbers of features transported.
57

---

Streling
Sourc
STOCAA 30 2015
Figure 43; FLUX Schnell I-step interventions; Example for edit category 9; "change style". Original prompt (target): "a photograph a clown with colorful hair", edit prompt (source): "a clown in pixel art style with colorful hair". Source and target refers to from where we extract features (source) and where we insert them (target). In this edit category we used features from the entire spatial grid. From this example it becomes clear that for this edit category we should select fewer features and probably lower strength. The y-labels indicate interventions strength and whether single- multi-layer interventions are used. The column titles indicate the intervention types and numbers of features transported.
58

---

A..3.
Drig.
A. 3.0
Cria
8.8.0
A.-8.0
DrG:
A.8.0
A-30
CFS
A.30
Dompiot
Desper
FIL
WEL
(a) down. 2.1
(b) mid.0
(c) up.
(d) up.
Figure 44: We visualize 6 features for down. 2. 2.1 (a), mid.0 0 (b), up. O, and up 1. We use three columns for each transformer block and three rows for each feature. For down. 2. and up. up.0.. 0. we visualize the samples from the top 5% quantile of activating dataset examples (middle) together a feature ablation (left) and a feature enhancement (right), and, activate the feature on the empty prompt with 7 0.5. 1,2 from left to right. For mid. and up O. we display three samples with ablation and enhancement. Captions are in Table 6 These results are from our first working SAE's with k- 10 and nt 5120.
59

---

Table 6: Prompts for the top 5% quantile examples in Fig.
44
Block
Feature
Prompt
down. 2.1
0
A tile folder with the word decument management on it. blue filled dividers.
A kitchen with an island and har stnols.
An unfinished with stools and a counter.
2
The Taj Mahal, The Taj Mahal, or a white marble building in Indin.
white marble building in India.
2
standing
each other.
3
Two men in suits hugging each other outside. An Forester whiskey
sitting top wooden Red roses and hearts on wonden table.
4
5
beaded brooch with pearls copper.
5
An image of a brooch with diamonds.
pedal an electronic
0
An engagement ring with blue sapphire and diamonds. The women's pink sneaker is shown.
0
1
white ceiling lan with three blades. cciling fan with three blades and a light.
1
ceiling dark
has wooden blades.
2
The black dress is made from knit and has metallic sleeves.
wearing
sports
2
The woman wearing striped swimsuit.
3
An old-fashioned photo frame with little girl on The woman sitting in her with her head down.
3
3
The contents of un emply bottle box.
An painting
uniform.
4
The model wears an off while sweatshirt with green panel. sky.
Liberty stands in
5
Cheese and crackers cutting board.
5
Two cufflinks with coins them.
Three pieces luggage shown blue.
up.0.0
glasses gold and silver designs. Three green wine glasses sitting next to each other.
0
0
New Year's with champagne, gold, and silver.
The birdhouse is made from wood and has brown reol.
The garage is white with red shutters, garages
porch An elegant white lace purse with gold clasp.
2
red handhag gold and pink and green floral-colored purse.
designs.
2
3
magazine rack with magazines on
3
The review page for this digital scrup.
3
The planner sticker kit shown with gold and black accessories. clock with
4
A silver watch with roman numerals on the face.
4
An automatic watch
silver dial. Four pieces of woeden furniture with blue and white designs.
5
5
The green chair is front white rug.
5
The wish chair with black seat.
u.
wooden toy kitchen includes bread, eggs, The office chair is brown and black. An serial view the white sand and turquoise water. An uerial view the beach and
0
1
I
2
The patriarch Ukrinne shown speaking to reportens,
German Chuncellor Merkel gestures she speaks the media.
3
Four pictures showing dogs wearing orange vests.
dogs are standing
ground to flowers.
4
man standing front of weoden wall.
4
blue mailbux sitting on top wooden floor.
5
The basehall players posing for photo.
5
The baseball players are holding up their trophics.
60

---

A..3.
Drig
3.0
A.80
8.8
A.-8.0
A.8.0
A-+30
Grip
A33
5119
201
(a) down.2.1
(b) mid.0
(c) up.
(d) 0p.
Figure 45: We visualize last 6 features for down 2. (a), mid. (b), up.0. 0. and up. 1. We use three columns for each transformer block and three rows for each feature. For down. 2. and up.0.1 0. visualize samples from the top 5% quantile of activating dataset examples (middle) together a feature ablation (left) and a feature enhancement (right), and, activate the feature on the empty prompt with 7 0.5, 1,2 from left to right. For mid. and up O. we display three samples with ablation and enhancement. Captions are in Table 7 These results are from our first working SAE's with k 10 and nt 5120.
61

---

Table 7: Prompts for the top % quantile examples in Fig. 45
Block
Feature
Prompt
down. 2.1
5114
Black and white Converse sacakers with the word black slar. Black white Converse sneakers the A woman holding
5115
photo of herself.
5115
air. The Nike Women's U.S. Soccer Team DRI-Fit Zip Top.
man holding up a teanis
5116
5116 The women's gray and orange half-zip sweatshirt.
largo group of people sitting front basketball
S117
Hockey players are playing in an urena with spectatiors.
plaid The different colors and sizes of t-shits..
black
shown.
5118
5119
of yarn on a white background.
5119
Two balls of colored wool are the white surface.
mid.0
building. Two men dressed in suits and ties are holding signs. signs.
People bolding signs
5114
5114
large group people bolding flags
5115
kitchen with white cabinets and blue stove.
5115
The kitchen clean and ready for us to use.
kitehen with white cabinets stainless
appliances.
S116
The steering wheel and dashboard in a car.
interior
dashboard controls.
5116
The dashboard and steering wheel in car.
5117
Three men are celebrating goal on the field.
5117
Red Bull racing gear standing Two men are posing for the camera at an event.
Two
each other.
5117
Someone holding up their polish pink black designs.
5118
The nail is very cute and looks great with marble.
stily nails
gold
5119
The Mighty Thor comic book.
5119
The camera showing its flash drive.
5119
iruck with bikes on the back parked next camper.
u.00.0
Acer laptop open ready
5114
The Lenovo S13 laptop is open and has an image of person jamping off the keyboard. laptop with the words Hosting Event on it.
5114
5115
and brown The horse leather oil being used to protect horses.
horse with black
5115
An painting on canvas
5116
The sun is shining brightly over Saturn.
foothall player throws
another
5116
Car door light logo sticker for Hyundai.
5117
An artistic black and silver sculpture with speakers.
5117
The pink brushes silting
each other.
5117
Four kings playing cards the hand.
lixing air conditionr.
5118
The black Land Rover is parked in front of large window.
screen TV mounted on the above fireplace.
5119
table with many different tools on it.
5119
camera with many different items including flash cards, lenses, and other accessories. The contents of open suitease and some clothes.
5119
u.
An Navajo nig with multicolored designs.
5114
The pillow made from old kilim.
5115
An image of noni juice with some fruits.
5115
bottle and glass the counter with green juice.
5116
Someone cleaning the shower with sponge.
skatehoard climbing a wall ropes.
5117
A man taking a selfie in front of some camera equipment.
5117
person holding up business card
cycle transportation.
5118
Two photos are placed top an open book.
5118
An open book with pictures of children and their parents.
5119
An engugement
with diamonis top.
5119
An oval raby and diamond ring.
62

---

A. -600
A -3.
O4g
A
A.60
A50
-30
Orig
A. 30
A. 6.0
(a) down.2.3
(b) up 0.1
Figure 46: We visualize 6 features for down. 2. 1 (a) and up. 0.1 (b). For each feature, we use columns showing ablations (left), activating examples (middle), enhancements (right) and rows with different samples from the top 5% quantile of activating examples. Captions are in Table B These results are from our first working SAE's with k and ny 5120.
63

---

One-
1.0
2.
3.3
4.
5.
OGG-
1.0
1.5
2,2.
A
A.30
5998
4559
4074
2202
(a) down. 2.
(b) up
Figure 47: We turn on the features from Fig. 46 on three unrelated prompts "a photo colorful model", "a cinematic shot of dog playing with ball", and "a cinematic shot of a classroom with excited students". These results are from our first working SAE's s with k = 10 and n, = 5120.
64

---

Table 8: Prompts for the top 5% quantile examples Fig. 46
Block
Feature
Prompt
holding something. Two cartoon pictures of the same man with his hands in his pockets.
carloon wearing
4998
4998
cartoon bear
pupple shirt and yellow shorts.
4074
An anime character with cat and dress.
4074
Two anime characters, one with white hair and the other with red eyes.
An book with
blue dresses.
2301 A man with white hair and red eyes holding a chuin. animated
hair
2301 The character standing with horns on his head.
Two men uniforms riding horses with swords. woman riding the back of brown horse.
Two jockeys on horses racing down the track. with floral designs
59
An old black vase with some design on it.
on StarCraft 2 is coming the Nintendo Wii. Overwatch is coming Xbox and PS3. The hero in Overwalch holding his weapon.
vase
u.
dog laying The woman posing for photo in her loopard print An animal print cube ottoman with brown white fur. white tiger with blue eyes standing bottle and liger are shown next to each other.
grass.
4955
4955
4977
4977
building Giraffes are standing in the grass near a vehicle.
tiger.
3718
giraffes standing
grass.
3718
giraffe standing next to ironing board.
lion roaring its teeth the snow.
lion sitting in the grass looking off into the distance. Two lions with flowers on their backs.
shining over
1093 Bride and groom in front of lake with sun flare.
milky sun shining brightly over the trees.
2165
The silhouctte of person riding a bike sunset.
2165
The Dark Knight rises from cave in Batman's 's poster.
2165
yellow sign with black design depicting tractor.
65

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are clearly stated in the abstract and introduction and sup- ported by quantitative and qualitative analyses.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have limitations paragraph in the discussion section. Additionally, in the supplementary material, we discuss the limitations, showcasing examples of unsuccessful SAE features.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
66

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our supplementary material contains all the information needed to reproduce the main experimental results. Additionally, we provide open- open-source implementations of our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
67

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code to reproduce the experiments is publicly available on github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/// httsssccpppbl//eeeeeeeeeeee htaab////uuieess/dduuuuiiiissss /at////odemiisssieeeeee spioioiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Full specification of settings and details the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Where applicable, plots report error bars and tables report standard deviation or standard errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
68

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Full details about compute resources are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurlPS Code of Ethics as uses publicly available data without involving human subjects or sensitive personal information.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The conclusion contains broad impact statement. Text-to-image models may be misused for generating deceptive content (c.g., deepfakes) and our work advances the development of interpretable, responsible, and human-centered AI systems.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
69

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., dentoyment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Both SDXL Turbo and FLUX Schnell which the focus of our investigation are relatively safe due to data filtering employed during all training stages. We are not aware of an immediate safety risk resulting from our work. Creating a high risk for misuse requires significant modifications by sophisticated user and we consider safeguarding against this threat-model beyond the scope of the current work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: SDXL and SDXL Turbo are under CreativeML Open RAIL++-M license and we created the remaining assets ourselves. Our dataset RIEBench is derived from PIEBench. which we prominently mention in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
70

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcooe com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide extensive description of all of our assets and artifacts in our paper and supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
71

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
72


Reviews:
[{"review_id": "S09KF2thvq", "reviewer": "Reviewer_SVh8", "summary": "Inspired by the application of sparse autoencoders in interpreting large language models, this work applied it to text-to-image diffusion models by training sparse autoencoders over the cross-attention blocks of one-step residual updates. This work found that training on one-step diffusion is able to explain most of the variance even in 4-step and 20-step diffusion. By using a SAM segmentation model, this work also showed that they can rank the importance of different found features and steer such features to control the generation process.", "questions": "1. For Figure 1 can you show the effect of changing the steering coefficient smoothly from negative to positive? Would the steering effect change smoothly as well?\n2. What's the benefit of training the sparse autoencoder on 1-step updates? If for most experiments you used 4-step updates, why not directly train it on 4-step updates? Is it an efficiency argument? (but I assume you can use 1/4 of the data to train on 4-step updates with the same cost?)", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Applying sparse autoencoders to text-to-image diffusion models is well-motivated to interpret features.\n2. Experiments show interesting specialization among blocks.\n3. The learned features can be used to steer the generation process.\n\nWeaknesses:\n1. It's not clear how stable the steering effect is. For example, if multiple samples were taken, would all show the same steering effect? Also, if the steering coefficient changes smoothly (from a large negative value to a large positive value), would the steering effect also change smoothly?\n2. It's not clear how well this method generalizes as experiments are mostly done on SDXL-Turbo.\n3. The results on Flux-schnell are not clear. Why does Figure 1 show \"the generalizability of our approach to recent Diffusion-Transformers\" (Line 146)? Why didn't you apply the same steering interventions as those applied to SDXL in each column (or did I misunderstand something here)?", "was_revised": true, "final_justification": "Thanks authors for their rebuttal. I think all my concerns have been answered, including the stability of the steering effect, the generalization ability, and also what Figure 1 means. Therefore, I'm recommending the acceptance of this paper and increasing my score to 5.", "rebuttal": "We thank the reviewer for their thoughtful evaluation of our work and for recognizing the motivation behind applying sparse autoencoders to text-to-image diffusion models. We appreciate the acknowledgment of our experimental findings showing interesting specialization among blocks and the demonstrated ability to use learned features for steering generation. We proceed by a shared response followed by addressing the individual concerns of the reviewer.\n\n## Shared response \n\n**Improved writing** We significantly improved their manuscript's organization and clarity through several key updates:\n\n- **Enhanced Structure & Contributions**\n    - Added clear paragraph headings for their four main contributions: SDLens (caching library), SAEPaint (visualization demo), RIEBench (editing benchmark), and findings on generalization across steps/architectures\n    - Updated abstract and section headers to match the new organization\n\n- **Content Consolidation & Clarification**\n    - **Section 3**: Merged spatial activation and sparse feature maps into a single \"feature interventions\" paragraph; added details on SDXL base interventions and classifier-free guidance handling\n    - **Section 4**: Renamed subsection 4.1 to better reflect the RIEBench benchmark; fixed a bug in equation (9); expanded details on benchmark construction and edit categories\n\n- **New Content & References**\n    - Added a new **Appendix B** specifically covering SDXL Base implementation details, including optimal strategies for feature addition during conditional/unconditional forward passes\n    - Systematically added appendix references throughout the paper to provide readers with additional technical details and examples\n    - Enhanced cross-referencing between sections and appendices (with updated lettering after inserting the new appendix)\n\n**Quantitative results for FLUX** In addition, to address the common concern of overly focusing our paper on SDXL Turbo, we performed an evaluation of FLUX Schnell on our representation-based editing benchmark (RIEBench). We will add these results to the camera-ready version of our paper in case of acceptance. For ease of presentation in the rebuttal format, we averaged across tasks here. We provide mean and Gaussian 95% confidence intervals of LPIPS between our edited-image and original-image CLIP similarity of our edited-image and the edit-prompt averaged over our 9 edit categories of which each contains about 30-50 examples.\n\n| Method | LPIPS Original | CLIP Text Edited |\n|--------|----------------|------------------|\n| sae_5_2 | 0.4436 ± 0.0272 | 0.2549 ± 0.0046 |\n| sae_5_1.5 | 0.3856 ± 0.0254 | 0.2534 ± 0.0046 |\n| sae_5_1.0 | 0.3085 ± 0.0225 | 0.2496 ± 0.0044 |\n| steering_0.4 | 0.4309 ± 0.0252 | 0.2475 ± 0.0041 |\n| steering_0.3 | 0.3712 ± 0.0217 | 0.2444 ± 0.0042 |\n| sae_5_0.5 | 0.1934 ± 0.0180 | 0.2433 ± 0.0043 |\n| steering_0.5 | 0.4686 ± 0.0269 | 0.2431 ± 0.0046 |\n| steering_1.0 | 0.5545 ± 0.0284 | 0.2243 ± 0.0052 |\n| baseline | 0.0000 ± 0.0000 | 0.2398 ± 0.0043 |\n\nAs can be seen, our FLUX SAE (hyperparameters: k=20, expansion factor=4) interventions, in which we transport 5 features from the mask within the edit-image into the mask of the original-image with varying strength, dominate the activation steering baseline (higher CLIP similarity with edit-prompt, while achieving lower perturbation of the original-image in terms of LPIPS distance). For reference we also provide a baseline in which we compute LPIPS between original image and itself (= 0) and CLIP similarity between the unmodified original-image and the edit-image.\n\n**Qualitative results and demo** It seems like reviewers did not find many of our provided qualitative results in our supplementary material as well as our installation-free demo in Appendix A. We highly recommend checking out the demo application if time permits it.\n\n## Individual concerns\n\nBelow, we address each of the raised concerns (prefixed with “**W**” when raised in weaknesses and with “**Q**” when raised in questions) individually. Responses are prefixed with “**R**”.\n\n**Q.1 & W.1** It's not clear how stable the steering effect is. For example, if multiple samples were taken, would all show the same steering effect? Also, if the steering coefficient changes smoothly (from a large negative value to a large positive value), would the steering effect also change smoothly?\n\n**R.** We demonstrate that the steering effect is both stable and smooth across multiple dimensions. Regarding smoothness, we provide comprehensive evidence in Figures 7 and 8 (App. B) for Flux and Figure 35 (App. G) for SDXL, which show smooth transitions as steering coefficients vary continuously from large negative to large positive values. Additionally, our demo application (App. A) allows interactive exploration of this smoothness property.\n\nFor stability across multiple samples, Figure 36 (App. G) provides clear evidence that the steering effects are consistent across different generations. Furthermore, Appendix F presents a detailed case study demonstrating that empty-prompt interventions produce consistent image changes, reinforcing the reliability of our approach. We encourage the reviewer to explore our demo application, which provides additional interactive evidence of both the stability and smoothness of steering effects.\n\n\n**Q.2** What's the benefit of training the sparse autoencoder on 1-step updates? If for most experiments you used 4-step updates, why not directly train it on 4-step updates? Is it an efficiency argument? (but I assume you can use 1/4 of the data to train on 4-step updates with the same cost?)\n\n**R.** Training on one-step updates provides multiple advantages beyond simple efficiency considerations. While computational efficiency is indeed a primary benefit, the deeper insight lies in the nature of feature activation patterns across timesteps.\nAs demonstrated in Figure 3, features activated across different timesteps exhibit remarkable consistency, with the quality of this consistency reflected in the explained variance metrics. Using 4-step generations would oversample correlated activations and repeat similar features/samples, effectively reducing the diversity of training samples within a fixed computational budget. This redundancy would not improve feature quality while increasing computational cost.\n\nMore fundamentally, our key insight is the successful decoupling of feature extraction methodology from feature injection during denoising. The surprising and practically valuable finding is that features extracted from 1-step activations effectively generalize to multi-step generation (4-steps for SDXL-Turbo, and even 20-25 steps for SDXL base). This demonstrates that our simpler training scheme is sufficient for extracting interpretable features, significantly simplifying the workflow for practitioners who need to train SAEs on diffusion models.\n\nThis finding challenges the intuitive assumption that training and inference should use matching step counts, instead revealing that the underlying feature representations are robust across different sampling strategies.\n\n\n**W.2** It's not clear how well this method generalizes as experiments are mostly done on SDXL-Turbo.\n\n**R.** Our work demonstrates generalizability across multiple model architectures and configurations. Specifically, we show that SDXL-Turbo features, without any additional training, successfully generalize to SDXL base models (as evidenced in Figures 1, 2, 3, and 36). This cross-model transferability is particularly noteworthy as it demonstrates the robustness of our learned representations.\nMore importantly, we replicate our SAE training methodology on FLUX, a state-of-the-art diffusion transformer-based text-to-image model, achieving successful results (Figures 1, 7, 8). This represents a significant architectural leap from the U-Net based SDXL models to transformer-based architectures. At the time of our initial submission and arXiv publication, we were the first work to train SAEs on intermediate results of the visual backbone in diffusion models. Furthermore, at the time of this NeurIPS submission, we were the first to achieve successful FLUX SAEs with effective feature steering capabilities.\n\n\n**W.3** The results on Flux-schnell are not clear. Why does Figure 1 show \"the generalizability of our approach to recent Diffusion-Transformers\" (Line 146)? Why didn't you apply the same steering interventions as those applied to SDXL in each column (or did I misunderstand something here)?\n\n**R.** We applied the same SAE training methodology to Flux-schnell activations, but the learned features naturally differ between architectures, and one-to-one feature matching is not always present. In some cases, achieving specific concepts requires combining multiple features (for example, creating a Santa Claus hat involves two distinct learned features working in conjunction).\nFigure 1 serves as a qualitative showcase of representative features learned by our SAEs on Flux-schnell, demonstrating successful causal interventions through feature steering. While the specific features differ from those learned on SDXL-Turbo (as expected given architectural differences), we present features with similar semantic content (blue color, \"evil\" characteristics, cartoon style) to illustrate conceptual correspondence. In contrast, features extracted from SDXL-Turbo one-step training successfully transfer to both SDXL-Turbo 4-step and SDXL base 25-step generation.\n\nThe generalizability of our approach refers to the effectiveness of training SAEs to extract interpretable features that enable causal interventions for activating high-level concepts across different architectures. This generalizability is non-trivial, particularly for Diffusion Transformer (DiT) architectures where the role of each layer and the causal relationship between directions in the representation space differ significantly from U-Net architectures."}, {"review_id": "vCxmHwZNZ2", "reviewer": "Reviewer_fSpm", "summary": "This paper investigates the application of sparse autoencoders (SAEs) to interpret and manipulate the intermediate representations of text-to-image diffusion models, specifically SDXL Turbo. By training SAEs on the residual updates of transformer blocks in SDXL Turbo’s one-step generation process, the authors successfully decompose complex hidden states into sparse and interpretable features. These features are shown to generalize across multiple denoising steps and even to other models like SDXL Base and FLUX Schnell. Using both qualitative visualizations and a quantitative benchmark adapted from PIEBench, the paper demonstrates that the learned features are not only interpretable but also causally influence the image generation process. Furthermore, the study reveals functional specialization among different transformer blocks, such as composition handling, object editing, and style refinement. This work represents a novel contribution to the mechanistic interpretability of diffusion models and establishes SAEs as a promising tool for understanding and editing their internal computations.", "questions": "1) While the proposed method is demonstrated on SDXL Turbo, which is a U-Net-based model, it remains unclear whether the approach can be applied to other one- or few-step generation models with different architectures, such as Diffusion Transformers. Have the authors considered how architectural differences may affect the applicability or reliability of SAE-based interpretability?\n\n2) The paper shows that SAE-learned features can influence the generated image, but it is not entirely clear what kinds of semantic or structural properties these features capture. Beyond the ability to manipulate generation, what specific insights into the model's behavior or representations were gained through this analysis?\n\n3) The caption for Figure 1 is somewhat minimal, especially concerning the Flux-schnell example. What do the numbers below the images represent, and how exactly does the image change as a result of the SAE intervention? It would be helpful to explain whether the SAE was trained using one-step intermediate features and then applied during a four-step or multi-step generation process for Flux-schnell, and how this intervention was performed.\n\n4) The authors state that the learned features “generalize” across different denoising schedules and models (from 1-step SDXL Turbo to 4-step and 20-step SDXL). Could the authors clarify what specifically is being generalized? Does this refer to the preservation of semantic meaning in SAE features, reconstruction quality, or transferability of causal influence?\n\n5) The analysis focuses solely on transformer blocks within the SDXL Turbo U-Net, even though the architecture also includes residual convolutional blocks. Given that convolutional and transformer features are known to exhibit different representational biases (e.g., texture vs. shape), it would be informative to know whether similar interpretability or control can be achieved through SAEs trained on convolutional features. Could the authors comment on why these components were not analyzed?\n\n6) The description of the generalization study (e.g., L133 and Figure 1) lacks sufficient detail. Could the authors explain more clearly how the visual examples were constructed? Specifically, how were SAE features trained in the one-step setting reused during multi-step generation, and how was the intervention performed?", "limitations": "\"No\" in the main paper.\n--\n- The analysis is limited to transformer blocks within the U-Net architecture, leaving out other critical components such as convolutional blocks, which may have distinct representational characteristics and contribute meaningfully to the generation process.  \n- Although the authors claim that the learned features generalize to multi-step generation and even to different models (e.g., SDXL Base, Flux-schnell), these claims are primarily supported by qualitative results, with limited quantitative evaluation or systematic comparison across architectures.   \n- The study is conducted on a relatively small number of U-Net blocks and features (e.g., four transformer blocks), raising the question of whether the observed interpretability and causal effects extend uniformly across the entire network.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n### First application of SAEs to diffusion models: \nThis is the first work to apply sparse autoencoders (SAEs) for interpretability in text-to-image diffusion models, filling a significant gap in the mechanistic understanding of such models, which has previously focused mainly on language models.\n\n### Causal and interpretable feature discovery:  \nThe SAE-learned features are shown to be semantically meaningful and causally affect the image generation process. Manipulating individual features results in consistent and interpretable changes in the generated images.\n\n### Feature-based image editing:  \nThe authors demonstrate that SAE features can be used as an effective image editing interface during generation, by activating or suppressing specific features to achieve desired visual modifications.\n\n### Block-level functional specialization:  \nThe analysis reveals that different transformer blocks in SDXL Turbo (e.g., down.2.1, up.0.0, up.0.1) specialize in different roles, such as layout composition, object detail, or style. This insight contributes to a finer understanding of the generation pipeline.\n\n# Weakness\n\n### Limited baseline & Insufficient evaluation on Diffusion Transformers:  \nThe study focuses primarily on SDXL Turbo, which, while compact and accessible, is not representative of the latest advancements in text-to-image generation. As newer models based on Diffusion Transformers—such as PixArt-series [1,2], SANA-series [3,4], or Stable Diffusion 3 [5]—have demonstrated strong performance, it would be valuable to assess whether the proposed SAE-based interpretability framework generalizes to these architectures. Although the authors include some qualitative examples on FLUX Schnell, the absence of quantitative results (e.g., explained variance, feature transport success) on these models limits the strength of the generalization claims. A more thorough evaluation on modern DiT-based models would significantly improve the impact and applicability of the approach. Including quantitative evaluations on one or more recent DiT-based models would significantly enhance the generality and relevance of the proposed approach, and help demonstrate whether the observed interpretability patterns hold across different model families.\n\n### Missing analysis of convolutional components in SDXL-Turbo:  \nThe paper focuses its analysis exclusively on transformer blocks within the U-Net of SDXL Turbo. However, the architecture also contains convolutional blocks, which play a crucial role in feature extraction and image synthesis. Understanding whether these convolutional components exhibit similarly interpretable or causally manipulable features is essential for a complete interpretability analysis of the model. As it stands, the findings may offer only a partial view of the internal mechanisms underlying image generation. Extending the SAE-based analysis to convolutional blocks would provide a more comprehensive understanding of the U-Net architecture and help reveal whether interpretability techniques can uncover complementary or distinct types of visual concepts encoded in different layer types.\n\n# Minor point\nThe expression “One Step is Enough” in the title may not accurately reflect the core contributions of the paper. It could give the impression that the work is focused on accelerating diffusion models or improving generation efficiency, which is not the main focus. Instead, the paper is centered on interpretability using sparse autoencoders. A more precise title would help align reader expectations with the actual scope of the study.\n\n\n\n---\n[1] Chen et al. PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR 2024.    \n[2] Chen et al. PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation. ECCV 2024.  \n[3] Xie et al. SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer. ICLR 2025.  \n[4] Xie et al. SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer. ICML 2025.  \n[5] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024.", "was_revised": true, "final_justification": "After reviewing the authors’ rebuttal and additional experiments, I have decided to maintain my original score of 4 (Borderline Accept). The paper presents a novel and timely contribution by applying sparse autoencoders (SAEs) to the interpretability of diffusion models, and I appreciate the authors’ detailed and thoughtful responses to my concerns. Below, I summarize my reasoning:\n- Evaluation on Diffusion Transformer (DiT) architectures:\nThe authors have extended their analysis to a DiT-based model (FLUX-Schnell), including both qualitative results and a commitment to include quantitative evaluation (via RIEBench) in the camera-ready version. While it would have been stronger to include more diverse DiT models and quantitative results in the current submission, the effort to test on a substantially different architecture is appreciated and provides partial evidence of generalizability.\n- Missing analysis of convolutional components:\nThe authors conducted new ablation experiments on ResNet blocks within SDXL-Turbo, showing that several convolutional layers have a substantial causal influence on the generated image. Although interpretability analysis using SAEs was not extended to these layers, the added results help paint a more complete picture of the U-Net architecture and directly address my concern.\n- Clarification of generalization claims and intervention methodology:\nThe rebuttal includes clear explanations of what is meant by “generalization” (semantic meaning, causal influence, and reconstruction quality), and provides a step-by-step description of the SAE intervention process during multi-step generation. These clarifications significantly improve the methodological transparency of the work.\n- Title and scope alignment:\nThe authors clarified the rationale behind the title “One Step is Enough” and acknowledged potential misunderstandings. They plan to revise the introduction to better communicate the intended message.\n\nOverall, the authors have addressed my main concerns to a reasonable extent. Some aspects—particularly broader quantitative evaluation on DiT models and analysis of convolutional layers using SAEs—remain limited, which is why I maintain a borderline rating. However, I believe this paper makes a meaningful contribution and opens up promising directions for future research on interpretability in generative models.", "rebuttal": "We sincerely thank the reviewer for their thorough evaluation and insightful feedback on our work. We are encouraged that the reviewer found our work to be the \"first application of SAEs to diffusion models,\" filling a \"significant gap,\" and appreciated the discovery of \"causal and interpretable features\" and the analysis of \"block-level functional specialization.\"\n\n## Shared response\nPlease find our response addressing shared concerns in our **response to reviewer SVh8**. \n\n## Individual concerns\nBelow, we address each of the raised concerns (prefixed with “**W**” when raised in weaknesses and with “**Q**” when raised in questions) individually. Responses are prefixed with “**R**”.\n\n**W.1.** Limited baseline & Insufficient evaluation on Diffusion Transformers \n\n**R.** We agree that a more thorough evaluation of our method on DiT architectures would significantly strengthen the paper. To this end we will include an evaluation of FLUX on our RIEBench benchmark into the camera-ready version of our paper (see shared response). \n\n**W.2.** Missing analysis of convolutional components in SDXL-Turbo \n\n**R.** The reviewer raises an excellent point regarding the role of other architectural components like ResNet blocks. Our initial focus was on the transformer blocks, given their established role in handling high-level semantic and compositional reasoning in generative models.\n\nHowever, to directly address the reviewer's comment, we ran an additional ablation study where we measured the causal impact of each ResNet block on the final image, similar to the analysis we performed for the attention blocks in our paper. The impact is quantified using the LPIPS distance between the original image and the image generated with a specific block ablated. The results are presented below (R stands for resnet and A for attention blocks):\n\n|Type|Block|Score|\n|-|-|-|\n|R|d0.r0|0.733|\n|R|d0.r1|0.415|\n|A|d1.a0|0.184|\n|A|d1.a1|0.160|\n|R|d1.r0|0.457|\n|R|d1.r1|0.305|\n|A|d2.a0|0.193|\n|A|d2.a1|0.512|\n|R|d2.r0|0.522|\n|R|d2.r1|0.269|\n|A|m.a0|0.345|\n|R|m.r0|0.239|\n|R|m.r1|0.131|\n|A|u0.a0|0.394|\n|A|u0.a1|0.520|\n|A|u0.a2|0.280|\n|R|u0.r0|0.347|\n|R|u0.r1|0.170|\n|R|u0.r2|0.156|\n|A|u1.a0|0.252|\n|A|u1.a1|0.217|\n|A|u1.a2|0.254|\n|R|u1.r0|0.198|\n|R|u1.r1|0.167|\n|R|u1.r2|0.201|\n|R|u2.r0|0.274|\n|R|u2.r1|0.702|\n|R|u2.r2|0.851|\n\n\nThis new analysis confirms that several ResNet blocks indeed have a substantial causal effect on the final output. Specifically, blocks such as down_blocks.0.resnets.1, down_blocks.1.resnets.0, and down_blocks.2.resnets.0. exhibit an impact comparable to or even greater than the most significant attention blocks we studied. There are resnet layers at the beginning and end of the unet that have an even larger LPIPS score but those destroy the entire image (ablating them results in pure noise), we think they encode/decode specifically to the noise and won’t yield interpretable features.\n\nWhile we find it highly interesting and agree that decomposing features from these ResNet blocks is a valuable research direction, we believe it is out of the scope of the current work. \n\n**W.3.** Minor point The expression “One Step is Enough” in the title may not accurately reflect the core contributions of the paper.\n\n**R.** The idea behind our title is to highlight a key and, we believe, surprising finding of our work: SAE features learned exclusively from the intermediate states of a one-step distilled model generalize remarkably well. They retain their semantic meaning and causal influence when applied in a multi-step generation process with the larger, non-distilled base models. We consider this a crucial result because it demonstrates that meaningful, interpretable features can be extracted with significantly lower computational resources (by training on the distilled model) and then effectively deployed to understand and edit more powerful, multi-step models. We will clarify this motivation in the introduction to better align reader expectations from the outset.\n\n\n**Q.1.** What about diffusion transformers?\n\n**R.** This is an important question that touches upon the generality of our findings. As mentioned in our response R.1, we have taken steps to explore the applicability of our method to other architectures. We have successfully trained SAEs on FLUX-Schnell, which is based on the Diffusion Transformer (DiT) architecture. Our results show that our method effectively learns interpretable and causally effective features in this different architectural paradigm as well. In our revised manuscript, we will present both qualitative examples and new quantitative results from our RIEBench benchmark on FLUX-Schnell to demonstrate that the approach is not limited to U-Net-based models.\n\n\n**Q.2.** What semantic and structural properties are captured by the features and what insights were gained? \n\n**R.** We thank the reviewer for this question and take this opportunity to clarify two central insights of our work.\nFirst, as detailed in Section 4.2 of our paper, our analysis reveals a strong functional specialization among different transformer blocks in SDXL. For example:\n- The down.2.1 block handles high-level composition and object creation. Interventions on this layer can generate entire objects corresponding to prompt concepts and have a global effect on the image.\n- The up.0.1 block specializes in style and texture. Its features can apply localized effects like animal textures, specific colors, or illumination without altering the underlying object structure.\n- The up.0.0 block acts as a detail-oriented layer. Its features are highly context-dependent and can be used to add or remove specific semantic properties of existing objects, such as removing a bow tie from a cat (as shown in our figures).\n- For FLUX-Schnell, we observed that the learned features tend to be more spatially localized, which is likely attributable to the higher spatial resolution of its latent space. We invite the reviewer to explore these behaviors interactively in our supplementary demo application.\nSecond, a key finding is the remarkable stability of these semantic features across different noise levels. Once a feature is activated during the denoising process, it tends to remain active. This stability is precisely what enables our SAEs, trained on one-step SDXL Turbo, to be transferred directly to the multi-step SDXL Base model without retraining, suggesting a shared semantic subspace between the distilled and base models.\n\n\n**Q.3.** The caption for Figure 1 is somewhat minimal.\n\n**R.** Thank you for pointing this out. We will revise the caption for Figure 1 and the corresponding methodology section to provide a more comprehensive explanation. The numbers below the images (e.g., #2301) are the indices of the specific features being activated from our learned SAE dictionary, which contains thousands of feature vectors. For the FLUX-Schnell example, the SAE was indeed trained on the intermediate features from its one-step generation process. The intervention was then performed during a multi-step generation to demonstrate the generalization of the feature's causal effect. We will make this process explicit in the revised manuscript.\n\n\n**Q.4.** Could the authors clarify what specifically is being generalized? \n\n**R.** The reviewer is correct in their interpretation. When we refer to \"generalization,\" we are referring to the preservation of all three aspects when moving from the one-step distilled model (where the SAEs were trained) to the multi-step base model:\n- Semantic Meaning: The features correspond to the same visual concepts (e.g., a specific style, object part, or texture) in both settings.\n- Causal Influence: Activating a specific feature produces a consistent and predictable change in the final image in both the one-step and multi-step generation processes.\n- Reconstruction Quality: For SAEs that achieve good reconstruction of the hidden state in the one-step model, they continue to be effective in the multi-step context.\nThis finding is intuitive, as the distilled model is initialized from and trained to mimic the base model, suggesting they share a significant portion of their representational structure. We will clarify this definition of \"generalization\" in the paper.\n\n**Q.5.** Missing analysis of convolutional components.\n\n**R.** See the response to W.2.\n\n**Q.6.** The description of the generalization study lacks sufficient detail. \n\n**R.** We apologize for the lack of clarity and thank the reviewer for highlighting it. As discussed in our responses R.6 and R.7, the process is as follows:\nTraining: An SAE is trained on a specific layer's hidden state activations, collected from numerous prompts using the one-step distilled model (e.g., SDXL Turbo).\nIntervention: During a multi-step generation process with the base model (e.g., SDXL Base over 20 steps), we intervene at a chosen denoising step. The intervention consists of adding a scaled version of a specific SAE feature vector (e.g., the 2301st feature from the dictionary) to the model's hidden state at the corresponding layer before it is passed to the next block.\nWe recognize this process is central to our claims and will revise the methodology section to include this detailed, step-by-step description to ensure it is perfectly clear and reproducible.\n\n**Regarding missing limitations**\n\n**R.** We thank the reviewer for this comment. We wish to clarify that a limitations section was indeed included in our original submission. As per NeurIPS guidelines, it is located within the checklist section, which appears after the references. We acknowledge that this placement can be easily overlooked. The key limitations discussed there include the points raised by the reviewer, namely the focus on transformer blocks (addressed in R.2) and the need for more quantitative evaluation on DiT models (addressed in R.1). We will ensure the limitations are more prominently signposted in the camera-ready version."}, {"review_id": "F5ViTP0hau", "reviewer": "Reviewer_siKW", "summary": "This paper explores the use of Sparse Autoencoders (SAEs) for interpreting and controlling text-to-image models, specifically SDXL Turbo. The authors train SAEs on updates made by transformer blocks within SDXL Turbo’s denoising U-net and find that they generalize well to different versions of SDXL models. The learned features from SAEs are interpretable, causally affect the generation process, and reveal block specialization. This is the first study to apply SAEs for interpretability in text-to-image models, demonstrating their potential for understanding and manipulating these models.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method that leverages a sparse autoencoder to interpret and control diffusion models is interesting.\n2. The paper is easy to understand. The experiments are detailed and comprehensive.\n\nWeakness:\n1. The generalizability of the proposed method in this paper needs further analysis. What conclusions would arise if the original SDXL model or other U-net-based diffusion models, such as SD 1.5, were used? Currently, many diffusion models are based on transformer architectures, so how can this type of analysis be applied to transformer-based models? Analyzing only the SDXL-Turbo model may not provide enough of a technical contribution to be widely applicable.\n2. The relationship with other existing works. The existing research[1] also draws similar conclusions, and the author needs to further explain the connections and differences between their work and the existing literature.\n3. Can the conclusions drawn from these observations further enhance the generation capabilities of diffusion models or expand their potential applications?\n4. Can these properties be leveraged to further enhance these capabilities during the training of diffusion models to improve the generative performance of the diffusion model?\n5. The authors should check the paper carefully before submmision to avoid the typo error.\n\n[1] P+: Extended Textual Conditioning in Text-to-Image Generation", "was_revised": true, "final_justification": "After reading the author's rebuttal, my concerns have been addressed. I raised my score to ba.", "rebuttal": "We thank the reviewer for their thoughtful evaluation of our work and for recognizing the interesting nature of our proposed method and the comprehensive experimental design. We appreciate the constructive feedback! The rebuttal is structured by first responding to common concerns raised by multiple reviewers and subsequently will address each of the raised concerns individually below.\n\n## Shared response \n\n**Improved writing** We significantly improved their manuscript's organization and clarity through several key updates:\n\n- **Enhanced Structure & Contributions**\n    - Added clear paragraph headings for their four main contributions: SDLens (caching library), SAEPaint (visualization demo), RIEBench (editing benchmark), and findings on generalization across steps/architectures\n    - Updated abstract and section headers to match the new organization\n\n- **Content Consolidation & Clarification**\n    - **Section 3**: Merged spatial activation and sparse feature maps into a single \"feature interventions\" paragraph; added details on SDXL base interventions and classifier-free guidance handling\n    - **Section 4**: Renamed subsection 4.1 to better reflect the RIEBench benchmark; fixed a bug in equation (9); expanded details on benchmark construction and edit categories\n\n- **New Content & References**\n    - Added a new **Appendix B** specifically covering SDXL Base implementation details, including optimal strategies for feature addition during conditional/unconditional forward passes\n    - Systematically added appendix references throughout the paper to provide readers with additional technical details and examples\n    - Enhanced cross-referencing between sections and appendices (with updated lettering after inserting the new appendix)\n\n**Quantitative results for FLUX** In addition, to address the common concern of overly focusing our paper on SDXL Turbo, we performed an evaluation of FLUX Schnell on our representation-based editing benchmark (RIEBench). We will add these results to the camera-ready version of our paper in case of acceptance. For ease of presentation in the rebuttal format, we averaged across tasks here. We provide mean and Gaussian 95% confidence intervals of LPIPS between our edited-image and original-image CLIP similarity of our edited-image and the edit-prompt averaged over our 9 edit categories of which each contains about 30-50 examples.\n\n| Method | LPIPS Original | CLIP Text Edited |\n|--------|----------------|------------------|\n| sae_5_2 | 0.4436 ± 0.0272 | 0.2549 ± 0.0046 |\n| sae_5_1.5 | 0.3856 ± 0.0254 | 0.2534 ± 0.0046 |\n| sae_5_1.0 | 0.3085 ± 0.0225 | 0.2496 ± 0.0044 |\n| steering_0.4 | 0.4309 ± 0.0252 | 0.2475 ± 0.0041 |\n| steering_0.3 | 0.3712 ± 0.0217 | 0.2444 ± 0.0042 |\n| sae_5_0.5 | 0.1934 ± 0.0180 | 0.2433 ± 0.0043 |\n| steering_0.5 | 0.4686 ± 0.0269 | 0.2431 ± 0.0046 |\n| steering_1.0 | 0.5545 ± 0.0284 | 0.2243 ± 0.0052 |\n| baseline | 0.0000 ± 0.0000 | 0.2398 ± 0.0043 |\n\nAs can be seen, our FLUX SAE (hyperparameters: k=20, expansion factor=4) interventions, in which we transport 5 features from the mask within the edit-image into the mask of the original-image with varying strength, dominate the activation steering baseline (higher CLIP similarity with edit-prompt, while achieving lower perturbation of the original-image in terms of LPIPS distance). For reference we also provide a baseline in which we compute LPIPS between original image and itself (= 0) and CLIP similarity between the unmodified original-image and the edit-image.\n\n**Qualitative results and demo** It seems like reviewers did not find many of our provided qualitative results in our supplementary material as well as our installation-free demo in Appendix A. We highly recommend checking out the demo application if time permits it.\n\n## Individual concerns\n\nBelow, we address each of the raised concerns (prefixed with “**W**” when raised in weaknesses and with “**Q**” when raised in questions) individually. Responses are prefixed with “**R**”.\n\n**W.1.** The generalizability of the proposed method in this paper needs further analysis. What conclusions would arise if the original SDXL model or other U-net-based diffusion models, such as SD 1.5, were used? Currently, many diffusion models are based on transformer architectures, so how can this type of analysis be applied to transformer-based models? Analyzing only the SDXL-Turbo model may not provide enough of a technical contribution to be widely applicable.\n\n**R.** We address the generalizability concern through multiple demonstrations in our work. First, in the paper and supplementary materials, we provide qualitative analysis for Flux-schnell (Figure 1 and Appendix B for more details), showing that our approach extends beyond SDXL-Turbo to transformer-based diffusion models. Additionally, we demonstrate that SAEs trained on SDXL-Turbo can be successfully applied to SDXL - the full multistep model - indicating cross-model transferability within the same architecture family. For our SDXL base results consider (Fig. 1 and Fig. 3 in the main paper, as well as Fig. 11, 17, 18, 19, 20, 21, 22, and 23 in the appendix provided in our supplementary material) or check out our installation-free demo application that we provide in Appendix A. While we acknowledge that broader evaluation across different architectures (SD 1.5, various transformer-based models) would strengthen our claims, our current results provide strong evidence for the method's generalizability across both distilled and full models, as well as different architectural paradigms (U-Net vs. transformer-based). Based on the fact that our SAE features transfer from SDXL Turbo to SDXL base without retraining, we are confident that they also transfer to other SDXL finetunes, of which there are about 10.000 on huggingface today 06/31/25.\n\n**W.2.** The relationship with other existing works. The existing research [P+: Extended Textual Conditioning in Text-to-Image Generation] also draws similar conclusions, and the author needs to further explain the connections and differences between their work and the existing literature.\n\n**R.** We discuss this work comprehensively in our \"Related Work\" section. Here is the key excerpt: \"In contrast to these prior works, our approach takes a fundamentally different perspective and does not rely on either handcrafted attributes or prompts. Instead, we introduce a new lens for analyzing SDXL Turbo's transformer blocks, which reveals specialization among the blocks as well. Interestingly, our findings on SDXL Turbo, which is a distilled, few-step diffusion model parallel [56]'s observations, which identify that composition precedes material and style. In SDXL Turbo, this progression occurs across the layers instead of the denoising timestamps.\" The key distinction is that our method provides interpretation without requiring predefined concepts or prompts, offering a more general and unbiased approach to understanding model behavior.\n\n\n**W.3.** Can the conclusions drawn from these observations further enhance the generation capabilities of diffusion models or expand their potential applications?\n\n**R.** Our approach significantly enhances image editing capabilities through interpretable feature manipulation. We envision a system where users have access to thousands of conceptual \"brushes,\" each corresponding to a specific SAE feature (color, texture, object, concept, detail, expression, illumination, shadow). Users can selectively apply these concepts to specific areas of an image, making targeted modifications with minimal impact on surrounding areas - a level of control not achievable through prompting alone. We provide a demonstration of this tool in Appendix A with an accompanying demo application. Furthermore, our method enables safety-oriented applications, such as modifying text-to-image diffusion models to adhere to safety standards by suppressing features associated with undesirable generations (NSFW, gore, etc.). The related work by Cywinski and Deja ([11] in our paper)  demonstrates this possibility for SD1.5, and our approach extends this capability with more interpretable and controllable mechanisms.\n\n\n**W.4.** Can these properties be leveraged to further enhance these capabilities during the training of diffusion models to improve the generative performance of the diffusion model?\n\n**R.** While our work primarily focuses on model inference and interpretation, these insights hold potential for constructing inherently interpretable architectures. We envision that our SAEs, particularly after labeling specific features, could enable novel distillation techniques that selectively transfer desirable features to student models while excluding undesirable ones. This could lead to more controllable and safer diffusion models from the training stage. However, we acknowledge that such applications represent a significant extension beyond our current scope and would require substantial additional research to validate these possibilities with certainty.\n\n\n**W.5.** The authors should check the paper carefully before submission to avoid the typo error.\n\n**R.** Thank you for bringing this to our attention and for helping maintain high scientific standards. We have thoroughly revised our submission (as mentioned in the **shared response**) and corrected all identified typographical errors. We greatly appreciate your diligence in pointing this out. If you have identified any specific typos, we would be grateful for your feedback so we can cross-reference them with our updated version to ensure complete accuracy.\n\n[11] Bartosz Cywi´ nski and Kamil Deja. Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders, 2025"}, {"review_id": "eXbfWPEexV", "reviewer": "Reviewer_uRaJ", "summary": "The proposed method provides an exploration over the learned features for SDXL and distilled diffusion/flow-matching models, where authors provide experiments mainly for SDXL-Turbo and demonstrate the extension on FLUX-schnell. Authors apply SAEs (sparse autoencoders) to the transformer layers of the models and provide analyses on the discovered representations over the features learned by the targeted transformer blocks. Following the SAE training, experiments analyze the explained variance by the targeted blocks across the timesteps. In addition to the variance analysis, authors also provide experiments on the significance of the discovered features on different types of edits, by utilizing a feature interpolation mechanism across the activations on the transformer blocks. Provided experiments show that features in specified blocks are responsible for certain generation patterns and utilized consistently for specified types of edits from PIE-Bench.", "questions": "- How do the authors decide on which transformer blocks to investigate?\n- Are the discovered representations be labeled with text instructions, or are they more ambiguous?\n- In the set of representations discovered, are there any overlaps, or does all of the representations discovered represent a distinct concepts?\n\nThese are the questions that I found to be beneficial for the clarity of the paper, and can improve the significance of the discovered representations. Answers to such questions in the rebuttal period will impact my judgement on the paper in a positive way.", "limitations": "While the authors provide suggestions on possible future work, they have no claims on the limitations if their work. As an example, are there any types of features that require combination of different transformer blocks, that cannot be identified with the current method. Also the usability of these features is another question. While editing results are reported, we have no clue on how successful these edits are or if the representations achieve disentangled edits overall. In the examples provided in Fig. 1, we see local changes such as skin color, but have no idea on more global features can be identified. Such limitations should be addressed in a possible revision.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper provides the first study on the application of SAEs for the explainability of features learned by diffusion models and their distilled counterparts.\n- The discovered features are investigated in terms of different types of editing tasks and their visualizations are provided to demonstrate their qualitative effectiveness (see Figure 1).\n- The paper introduces a feature injection scheme to effectively use them for editing tasks\n\n**Weaknesses**\n- The organization of the paper should be improved. The narrative of the paper seems disconnected, which makes it hard to understand.As an example for this, the paper first explains the SAEs trained and the blocks that are selected for it. Then, in the analysis section, the example of feature transport is given. The connection here should be improved for writing, in the current form it feels disconnected (like saying that this is used to transport SAE features in the experiments etc.)\n- The readability of the figures should be improved significantly. As an example for this, Figure 5 reports the impact on the editing performance. However, with the labels for SAE features the plot is almost unreadable. The authors shuld consider revising accordingly.\n- In lines 117 and 118, the authors say that they select 4 layers for their analysis. However, there is no motivation on how they were selected. The ,methodology should either refer to previous studies on the investihgated blocks or provide a convincing reason on why these methods are investigated.\n- While the authors provide editing results with discovered representations, their quality is uncertain. For example, we have no idea on if the representations discovered are repetitve and how unique they are. Furthermore, to show the effectiveness of such representations, the authors should provide qualitative editing results for the categories that the edits are reported.\n- To demonstrate if the discovered representations are beneficial for image editing task, a comparison with existing editing methods is essential (for distilled methods, a comparison with [1] can be provided). Conversely, if edits that cannot be performed with the existing methods can be performed with the proposed method, then this should be stated explicitly.\n- As we have no explicit information on which types of representations are discovered, the limitations of the method is ambiguous. An analysis on the discovered representations will improve the significance of the methodology.\n- With the current state of the paper, it appears that the authors get an existing interpretability method (SAEs) and plug it into an existing method. While this is okay for the explanation of the representations, it makes the novelty of the paper questionable. The authors should provide details on how the adaptation is accomplished (like the selection of the features, how it differs from applying SAEs to LLMs) to further demonstrate the contribution of the paper (or explain their analysis in detail).\n\n[1] Deutch et. al. \"TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models\", https://arxiv.org/abs/2408.00735", "was_revised": true, "final_justification": "My concerns regarding the novelty of the method and the types of representations discovered has been addressed by the rebuttal of the authors. After reviewing the responses of the other reviewers, I am leaning towards positive.", "rebuttal": "Thank you for your thoughtful and constructive feedback on our paper. We appreciate the time you invested in reviewing our work and the specific suggestions you provided to improve the clarity, organization, and significance of our contributions. We have carefully considered each of your concerns and will address them individually below. Your feedback has been instrumental in helping us strengthen both the presentation and the technical content of our work.\n\n## Shared response\n\nWe recommend having a look at our shared response (contained in our **responses to reviewer SVh8 and siKW**) for our new quantitative results on FLUX.\n\n## Individual concerns\n\n**Q.1. / W.3.** How do the authors decide on which transformer blocks to investigate?\n\n**R.** We provide detailed analysis in Appendix C \"Finding causally influential transformer blocks.\" We performed ablation studies on each individual transformer block and found that only these 4 blocks have significant effects on generated images when modified. This selection is also supported by prior work (particularly the P+ study) showing that blocks from the center of the U-Net architecture are responsible for high-level concepts like objects, composition, and colors. Also, in the scope of our rebuttal for **fSpm** (their W.2.) we provided additional results for resnet blocks.\n\n\n**Q.2.** Are the discovered representations labeled with text instructions, or are they more ambiguous?\n\n**R.** We used a comprehensive annotation pipeline to label features of one of the blocks (down.2.1), as detailed in Appendix H.1 with examples in Appendix I.3. To annotate features, we prompted GPT-4o with series of images including: (1) images where the feature is inactive, (2) progression of images with increasing average activation values, and (3) images with the highest average activation values. This approach allows us to generate meaningful text labels for discovered representations, making them interpretable and usable for targeted editing applications.\n\n\n**Q.3** In the set of representations discovered, are there any overlaps, or do all of the representations discovered represent distinct concepts?\n\n**R.** Learning features with top-k SAEs involves inherent trade-offs in choosing expansion factors and numbers of nonzero coefficients, which affect feature overlap properties. We observe some degree of feature splitting (where single concepts get divided into multiple features, e.g., separate features for cat ears, eyes, mouth, fur), but this rarely occurs with our chosen hyperparameters for our demo application (k=10, expansion factor=4). We also observe features for similar concepts across different blocks, but these are rarely redundant functionally. Instead, different blocks specialize in different aspects: down.2.1 may place a \"cat\" in the scene, up.0.0 may modulate details like facial expressions, and up.0.1 may determine color and texture details. Additionally, we find significant feature universality between FLUX and SDXL models, as demonstrated in Figure 1 and our demo applications (Appendix A).\n\n\n**W.1.** The organization and presentation of the paper should be improved. \n\n**R.** Since we found this concern highly important, we provide an overview of the changes to our manuscript that we have performed since the submission in the **shared response**.\n\n\n**W.2.** The readability of the figures should be improved significantly. \n\n**R.** We acknowledge this important concern and will significantly improve the figure readability for the camera-ready version. For Figure 5 specifically, we will implement a cleaner visualization approach that conveys the same information about the Pareto dominance of SAEs versus other methods without overwhelming the reader. Our planned improvements include: color-coding the number of features transported using a clear legend, using different symbols for different steering strengths, and potentially splitting complex plots into multiple subplots for better clarity. We are committed to ensuring all figures are easily interpretable and support the paper's narrative effectively.\n\n\n**W.4.** The quality of the editing results via the discovered features is uncertain. \n\n**R.** We provide comprehensive case studies in our appendices that address these concerns. Specifically, Appendix A contains an installation free demo application to try out our brush tool. Appendix B contains additional results for FLUX. Appendix D contains qualitative results for applying interventions during different ranges of the multi-step setting. Appendix E contains qualitative examples for each edit category in RIEBench. Appendix F presents \"Case study: Most active features on a prompt,\" Appendix G covers \"Case study: random features,\" and Appendix H provides \"Quantitative evaluation of the roles of the blocks.\" These studies demonstrate the quality, uniqueness and diversity of discovered representations. Our analysis shows that while some degree of feature overlap exists (particularly feature splitting with higher expansion factors), the features we discovered with our chosen hyperparameters for the demo application (k=10, expansion factor=4) are largely distinct and functionally non-redundant across different blocks. \n\n\n**W.5.** To demonstrate if the discovered representations are beneficial for image editing tasks, a comparison with existing editing methods is essential.\n\n**R.** To the best of our knowledge, the suggested related works like TurboEdit don't fall within the same category of method. Our interventions currently can only modify images while they are generated, whereas, TurboEdit is meant for editing real images. We are not aware of related works in the same category. Our SAE features and brush tools enable customization of text-to-image generation in ways that are not achievable through prompting-based techniques. While existing methods could potentially be adapted through masking restrictions, the diversity of SAE features learned in an unsupervised manner makes it challenging to match their effects with competing approaches using similar resources. For example, we can locally apply animal skin/fur textures onto human faces without perturbing the underlying facial structure—our brush tools can apply 5 different textures as demonstrated in our supplementary materials. Additionally, our multi-timestep approach with timed-hooks that act only on subsets of timesteps provides novel means of trading off similarity to the original image with edit strength. Real image editing is also possible when combining our SAE features with diffusion inversion techniques, though this area requires further exploration beyond the current scope.\n\n\n**W.6.** As we have no explicit information on which types of representations are discovered, the limitations of the method is ambiguous. An analysis on the discovered representations will improve the significance of the methodology.\n\n**R.** We provide detailed analysis of discovered representations throughout our work (Section 4.2., Appendix F, G, and H). Our key findings include: (1) different transformer blocks specialize in different aspects of image generation (e.g., down.2.1 places objects in scenes, up.0.0 modulates details like facial expressions, up.0.1 determines colors and textures); (2) features remain stable across timesteps, enabling one-step learning to transfer to multi-step generation; (3) we observe feature universality across different model architectures (SDXL and FLUX), as demonstrated in Figure 1 and our demo applications. The discovered representations span various semantic levels from high-level concepts to fine-grained textural details, providing comprehensive control over generation aspects.\n\n\n**W.7.** The novelty of the paper is questionable.\n\n**R.** Our work presents several significant novel contributions beyond simply applying existing methods. Key innovations include: (1) the crucial finding that one-step activation patterns can be applied to multi-step generation without retraining, which is essential for both understanding representation spaces and practical applications; (2) addressing unique challenges in the denoising process that differ substantially from LLM applications, including how activations evolve across different forward passes; (3) developing empty prompt interventions and specialized techniques for classifier-free guidance models; (4) creating image editing methods with semantic segmentation masks and brush tools that are unique to the visual modality and address spatial localization challenges not present in text applications. Furthermore, when applying SAEs to DiTs like FLUX, we found it of key importance to distinguish our \"feature extraction\" phase from \"feature injection\", which is in stark contrast to how things are done in LLMs. We find that those two are not strictly connected, and our injection methods represent novel contributions to visual generation control.\n\n\n**Regarding missing limitations** \n\n**R.** We acknowledge this important point and will add a comprehensive limitations section to our revised paper. Key limitations include: (1) our current method focuses on individual block analysis and may miss features that require combinations of multiple transformer blocks; (2) the success rate and disentanglement quality of edits varies depending on the feature type and target image content; (3) while we demonstrate local changes effectively (like skin color modifications), the identification and control of more global compositional features remains challenging; (4) our approach is currently limited to the specific architectures we studied (SDXL and FLUX), and generalization to other diffusion models requires further validation; (5) the hyperparameter choices for SAE training (expansion factor, sparsity level) significantly impact the types and granularity of discovered features, requiring careful tuning for optimal results."}]

Author rebuttal:

Meta-review: This paper focuses on the lack of interpretability of standard text-to-image diffusion models. Inspired by the use of sparse autoencoders (SAEs) in the context of LLMs to decompose intermediate representations sparse sums of interpretable features, this work proposes to train SAEs on the residual updates of transformer blocks in SDXL Turbo. This is first shown on the SDXL Turbo's one-step generation process. Without additional training, this 1-step trained model also applies to the 4-step and the 20-step cases.

This work has several strengths: good motivation for applying SAEs to text-to-image diffusion models, causal and interpretable features, generalization of 1-step to 4-step and 20-step model variants, feature-based image editing. After the reviewer-author and the reviewers-AC discussions, all the reviewers confirmed that their initial questions were clearly addressed by the authors. The main remaining question was on the organization and clarity of the paper. This can be addressed in a final revision of the paper.

The strengths, especially addressing an important problem of interpretability of large generative models such as text-to-image diffusion models, clearly outweigh the other minor negative points on organization and clarity of the paper. In view of this, the AC recommends the paper for acceptance. Addressing the remaining issue and including all the clarifications provided in the rebuttal and discussion phases will make for a more complete paper.
Decision: Accept (poster)