ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Authors: Alec Helbling
Tuna Han Salih Meral
Benjamin Hoover
Pinar Yanardag
Duen Horng Chau
Venue: ICML 2025 oral (2025)
Keywords: diffusion
interpretability
transformers
representation learning
mechanistic interpretability
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We introduce a method for interpreting the representations of diffusion transformers by producing saliency maps of textual concepts.

Abstract: Do the rich representations of multi-modal diffusion transformers (DiTs) exhibit unique properties that enhance their interpretability? We introduce ConceptAttention, a novel method that leverages the expressive power of DiT attention layers to generate high-quality saliency maps that precisely locate textual concepts within images. Without requiring additional training, ConceptAttention repurposes the parameters of DiT attention layers to produce highly contextualized *concept embeddings*, contributing the major discovery that performing linear projections in the output space of DiT attention layers yields significantly sharper saliency maps compared to commonly used cross-attention maps. ConceptAttention even achieves state-of-the-art performance on zero-shot image segmentation benchmarks, outperforming 15 other zero-shot interpretability methods on the ImageNet-Segmentation dataset. ConceptAttention works for popular image models and even seamlessly generalizes to video generation. Our work contributes the first evidence that the representations of multi-modal DiTs are highly transferable to vision tasks like segmentation.

ConceptAttention:
Diffusion Transformers Learn Highly Interpretable Features
Alec Helbling'
Tuna Han Salih Meral Benjamin Hoover
Pinar Yanardag" 2
Duen Horng (Polo) Chau'
Abstract
dragon hill."
Do the rich representations of multi-modal dif- fusion transformers (DiTs) exhibit unique prop- erties that enhance their interpretability? We in- troduce CONCEPTATTENTION a novel method that leverages the expressive power of DiT at- tention layers to generate high-quality saliency maps that precisely locate textual concepts within images. Without requiring additional training. CONCEPTATTENTION repurposes the parameters of DiT attention layers to produce highly con- textualized concept embeddings, contributing the major discovery that performing linear projec- tions in the output space of DiT attention layers yields significantly sharper saliency maps com- pared to commonly used cross-attention maps. CONCEPTATTENTION even achieves state-of-the- art performance zero-shot image segmenta- tion benchmarks, outperforming 15 other zero- shot interpretability methods on the ImageNet- Segmentation dataset. CONCEPTATTENTIO works for popular image models and even seam- lessly generalizes to video generation. Our work contributes the first evidence that the representa- tions of multi multi-moda modal DiTs are highly transferable to vision tasks like segmentation.
Multi- Mudai-Modal
Attention
CONCEPT ATTENTION
dragon
rock
sun
clouds
Cross Attention (FLUX)
dragon
rock
clouds
DAAM (SDXL)
dragon
rock
clouds
TextSpan (CLIP)
dragon
rock
clouds
Figure 1. CONCEPTATTENTION produces saliency maps that precisely localize the presence of textual concepts in images. We compare Flux raw cross attention, DAAM (Tang et al., 2022) with SDXL, and TextSpan (Gandelsman et al., 2024) for CLIP.
1. Introduction
Diffusion models have recently gained widespread popular- ity, emerging as the state-of- of-the-art approach for variety of generative tasks, particularly text-to-image synthesis (Rom- bach et al., 2022). These models transform random noise into photorealistic images guided by textual descriptions, achieving unprecedented fidelity and detail. Despite the impressive generative capabilities of diffusion models, our understanding of their internal mechanisms remains lim- ited. Diffusion models operate black boxes, where the
relationships between input prompts and generated outputs visible, but the decision-makinn processes that connect them are hidden from human understanding.
Existing work on interpreting T2I models has predominantly focused on UNet-based architectures (Podell et al., 2023;
Rombach et al., 2022), which utilize shallow cross-attention mechanisms between prompt embeddings and image patch representations. UNet cross atlention maps can produce high- ligh-fideityy saliency maps that predict the location of tex- tual concepts (Tang et al., 2022) and have found numer- ous applications in tasks like image editing (Hertz et al., 2022; Chefer et al., 2023). However, the interpretability
'Georgia Tech Viirginia Tech 'IBM Research. Correspondence to: Alec Helbling <alechelbling @ gatech.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Code: alechelbl ing www.inn.n......ooommomm.o....e.o com//ooneptAttentt

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Multi-modal DIT
Multi-modal DiT with CONCEPTATTENTION
MM-ATTN
MM-ATTN
Prompt
"A dragon on a hill."
Tokens
Prompt
dragon on a hill."
Tokens
MM--A AMM-ATTN
MM- ATTN
Image Tokens
MM- ATTN
MM- MM-ATTN ATTN
Image Tokans
Saliency Map
Concapt
dragon
**.
rock
A Iteratively updates prompt and image tokens
B Also process concept tokens for interpretability
Figure CONCEPTATTENTION augments multi-modal DiTss with a sequence of concept embeddings that can be used to produce saliency maps. (Left) An unmodified multi-modal attention (MMATTN) layer processes both prompt and image tokens. (Right) CONCEPIAITENTION augments these layers without impacting the image appearance to create a set of contextualized concept tokens.
of more recent multi-modal diffusion transformers (DiTs) remains underexplored. DiT-based models have recently replaced UNets (Ronneberger et al., 2015) as the state-of- the-art architecture for image generation, with models such Flux (Labs, 2023) and SD3 (Esser et al., 2024) achieving breakthroughs in text-to-image generation. The rapid ad- vancement and enhanced capabilities of DiT-based models highlight the critical importance of methods that improve their interpretability, transparency, and safety.
text-to-image diffusion transformers. Our method requires no additional training, by leveraging the rep- resentations of multi-modal DiTs to generate highly interpretable saliency maps that depict the presence of arbitrary textual concepts (e.g. "dragon", "sky", etc.) images (as shown in Figure 1).
The novel discovery that the output vectors of at- tention operations produce higher-quality saliency maps than cross attentions. CONCEPTATTENTION repurposes the parameters of DiT attention layers produce rich textual embeddings corresponding dif- ferent concepts, something that is uniquely enabled by multi-modal DIT architectures. By performing linear projections between these concept embeddings and im- age patch representations in the attention output space we can produce high quality saliency maps.
In this work, we propose CONCEPTAITENTION a novel method that leverages the representations of multi-modal DiTs to produce high-fidelity saliency maps that localize tex- tual concepts within images. Our method provides insight into the rich semantics of DiT representations. CONCEPTAT- TENTION is lightweight and requires no additional training, instead repurposes the existing parameters of DiT atten- tion layers. CONCEPTAITENTION works by producing a set of rich contextualized text embeddings each correspond- ing to visual concepts (e.g. "dragon", "sun"). By linearly projecting these concept embeddings and the image we produce rich saliency maps that are even higher quality than commonly used cross attention maps.
CONCEPTATTENTION achieves state-of- ootte-of-thh-attt per- formance in zero-shot segmentation on benchmarks like ImageNet Segmentation and Pascal VOC across multiple DiT architectures. We achieve superior per- formance to diverse set of zero-shot interpretabil- ity methods based on various foundation models like CLIP, DINO, and UNet-based diffusion models; this highlights the potential for the representations of DiTs to transfer to important downstream vision tasks like segmentation. We replicate our results quantitatively on both Flux and Stable Diffusion 3.5 Turbo.
We evaluate the efficacy of CONCEPTAITENTION in zero- shot semantic segmentation task on real world images. We compare our interpretative maps against annotated segmen- tations measure the accuracy and relevance of the attri- butions generated by our method. Our experiments and extensive comparisons demonstrate that CONCEPTATTEN- TION provides valuable insights into the inner workings of these otherwise complex black-box models. By explaining the meaning of the representations of generative models our method paves the way for advancements in interpretability, controllability, and trust in generative AI systems.
CONCEPTATTENTION works with video genera- tion DÎT model. Additionally, we demonstrate qual- itatively that CONCEPTAITENTION seamlessly gen- eralizes to the CogVideox (Yang et al., 2025) video generation MMDiT model, producing higher-quality saliency maps than native cross attention maps.
In summary, contribute:
CONCEPTATTENTION, a method for interpreting
2

---

ConceptAttention: Diffusson Transformers Learn Highly Interpretable Features
flowers
suit
sky
face
tree
grass
A man with flowers.
ballãon
sun
sky
mountain
clouds
water
A hot air balloon over lake.
Figure 3. CONCEPTATTENTION can generate high-quality saliency maps for multiple concepts simultaneously. Additionally, our approach is not restricted to concepts in the prompt vocabulary.
2. Related Work
of our saliency maps and the transferability of the repre- sentations of multi- DiT architectures to downstream vision tasks. This task also provides a good setting to com- pare to a variety of other interpretability methods for various foundation model architectures like CLIP (Radford et al., 2021), DINO (Caron et al., 2021), and diffusion models.
Diffusion Model Interpretabilityy A fair amount of ex- isting work attempts to interpret diffusion models. Some works investigate diffusion models from an analytic lens (Kadkhodaie et al., 2024; Wang et al., 2024), attempting to understand how diffusion models geometrically model the manifold of data. Other works attempt to understand how models memorize images (Carlini et al., 2023). An increas- ing body of work attempts to repurpose the representations of diffusion models for various tasks like classification (Li et al., 2023a), segmentation (Karazija et al., 2024), and even robotic control (Gupta et al., 2024). However, most relevant to our work is the substantial body of methods investigating how the representations of the neural network architectures underpinning diffusion can be used garner insight into how these models work, steer their behavior, and improve their safety.
A variety of works train models from scratch for the task of image segmentation (Amit et al., 2022; Karazija et al., 2024) or attempt to fine-tune pretrained models (Baranchuk et al., 2022). Another line of work leverages diffusion models to generate synthetic data that can be used to train segmenta- tion models that transfer zero-shot to new classes (Li et al., 2023b). While effective, these methods are training- based and thus do not provide as much insight into the representa- tions of existing text-to-image generation models, which is the key motivation behind CONCEPTATTENTIOS
A significant body of work attempts to improve the inter- pretability of CLIP vision transformers (ViTs) (Dosovitskiy et al., 2021). The authors of (Chefer et al., 2021) develop method for generating saliency maps for ViT models, and they introduce an evaluation protocol for assessing the ef- fectiveness of these saliency maps. This evaluation protocol centers around the laumin et al., 2014), and we extend this evaluation to the PascalVOC dataset (Everingham et al., 2015). They com- pare to variety of zero-shot interpretability methods like GradCAM (Selvaraju al., 2019), Layerwise- Layerwise-Relevance Relevance Propagation (Binder et al., 2016), raw attentions, and the Rollout method (Abnar & Zuidema, 2020). The authors of (Gandelsman et al., 2024) demonstrate an approach to ex- pressing image patches in terms of textual concepts. We also compare our approach to shot diffusion based methods (Tang et al., 2022; Wang al., 2024) and the self-attention maps of DINO ViT models (Caron et al., 2021).
Numerous papers have observed that the cross attention mechanisms of UNet-based diffusion models like Stable Diffusion (Rombach et al., 2022) and SDXL (Podell al., 2023) can produce interpretable saliency maps of textual concepts (Tang et al., 2022). Cross attention maps used in variety of image editing tasks like producing masks that localize objects of interest to edit (Dalva et al., 2024), con- trolling the layout of images (Chen et al., 2023; Epstein et al., 2023), altering the appearance of an image but retaining its layout (Hertz et al., 2022), and even generating synthetic data to train instruction based editing models (Brooks et al., 2023). Other works observe that performing interventions on cross attention maps can improve the faithfulness of im- ages to prompts by ensuring attributes are assigned to the correct objects (Meral et al., 2024; Chefer et al., 2023). Ad- ditionally, it has been observed that self-attention layers of diffusion models encode useful information about the layout of images (Liu al., 2024).
dataset (Guil-
Another line of work performs unsupervised segmentation without any class or text conditioning by performing cluster- ing of the embeddings of models (Cho et al., 2021; Hamilton et al., 2022; Tian et al., 2024). Despite not producing class predictions, these models are often evaluated on semantic
Zero-shot Image Segmentation In this work, we evalu- ate CONCEPTATTENTION on the task of zero-shot image segmentation, which a natural way to assess the accuracy
3

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
CONCEPT ATTENTION
Cross Attn (Flux)
DAAM (SDXL)
TextSpan (CLIP)
Transinterp (CLIP)
Rollout (CLIP)
LRP (CLIP)
Last Layer LRP Raw Attn (CLIP)
GradCAM (CLIP)
Ground Truth
(CLIP)
Preds
Raw
Figure 4. CONCEPIATTENTION produces higher fidelity raw scores and saliency maps than alternative methods, sometimes surpassing in quality the ground truth saliency map provided by the ImageNet-Segmennation task. Top row shows the soft predictions of each method and the bottom shows the binarized predictions.
segmentation datasets by using approaches like Hungar- ian matching (Kuhn, 1955) to pair unlabeled segmentation predictions with the best matching ones in multi-class semantic segmentation dataset. In contrast, CONCEPTAT- TENTION enables text conditioning so we do not compare to this family of methods. We also don't 't compare to models like SAM (Kirillov et al., 2023; Ravi et al., 2024) as it is trained on a large scale dataset.
are two key classes of layers: one that keeps separate resid- ual streams for each modality and one that uses a single stream. In this work, we take advantage of the properties of these dual stream layers, which we refer to multi-modal attention layers (MMATTNS).
The input to given layer is sequence of image patch representations x E fghxwxd and prompt token embeddings p E Rixd The initial prompt embeddings at the beginning of the network are formed by taking the T5 (Raffel et al., 2023) embeddings of the prompt tokens.
3. Preliminaries
3.1. Rectified-Flow Models for Image Generation
Following (Peebles & Xie, 2023), each MMATTN layer leverages a set of adaptive layer norm modulation layers (Xu et al., 2019), conditioned on the time-step and global CLIP vector. An adaptive layernorm operation is applied to the input image and text embeddings. The final modulated outputs are then residually added back to the original input. Notably, the image and text modalities kept in separate residual streams. The exact details of this operation omitted for brevity.
Flux and Stable Diffusion 3 leverage multi-modal DiTs that are trained to parameterize rectified flow models. Through- out this paper we may refer to rectified flow models as diffusion models for convenience. These models attempt to generate realistic images from noise that correspond to given text prompts. Flow based models (Lipman et al., 2023) attempt map sample 1 from a noise distribution P1. typically P1 (00 I), sample Ty in the data distri- bution. Rectified flows (Liu et al., 2022) attempt to learn ODEs that follow straight paths between the and Pı, i.c.
The key workhorse in MMATTN layers is the familiar multi- head self attention operation. The prompt and image embed- dings have separate learned key, value, and query projection matrices which we refer to Qu,Qu, Va for images and Kp Qp Vp for text. The keys, queries, and values for both modalities collectively denoted qep. kap, and Uxp, where for example kep = [Kzz,...., operation is then performed
2 = (1-t)xa tc,c 10,
(1)
Flux and SD3 are trained using conditional flow matching objective which can be expressed conveniently as
A self attention
1 Boneett
- =
-
Og Op = coftmax((uppe) Uzp
Here we refer to 0g and op as the attention output vectors. Another linear layer is then applied to these outputs and added to a separate residual streams weighted according to the output of the modulation layer. This gives us updated embeddings L++1 and p which are given as input the next layer.
where Xr corresponds to signal-to-noiss ratio and wi is time dependent- weightingg factor. Above (0 t) is param- eterized by a multi-modal diffusion transformer network. The architecture of this model and it's properties is of pri- mary interest in this work.
3.2. The Anatomy of a Multi-modal DiT Layer
4. Methods
Multi- mudti-modal DiTs like Flux and Stable Diffusion 3 lever- age modal attention layers (MMATTN) that process a combination of textual tokens and image patches. There
We introduce CONCEPTATTENTION method for gener- ating high quality saliency maps depicting the location of
4

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
(a) Multi-modal Attention
the image patches respectively. We can concatenate the image and concept keys get
Image Keys
Prompt Keys
Kize [K KaTa, K,01 ... Kpt]
(5)
Image Queries
Image Self Attention
Prompt->
Cross Atlention
Image and the image and concept values to get
Values
Output
Vae = Va VaJn, V1 ..., [
(6)
Prompt Quanes
Prompt
Prompt Output
Pragpt
Prompt Selt Attention
Attention
We can then perform the following attention operation
O.: = softmax (softmax(qe Uze
(b) Concept Attention
Image Keys
Concapt Keys
which produces set of concept oulput embeddings.
Image Values
Notice, that instead of just performing a cross attention (i.e. softmax(qck2] )u our approach leverages both cross attention from the image patches the concepts and self at- tention among the concepts. We found that performing both operations improves performance on downstream evaluation tasks like segmentation (See Table 4). We hypothesize this is because it allows the concept embeddings to repel from each other, avoiding redundancy between concepts.
Concapt
Image Concept
Concept Self Attention
Concept Output
Cross Attention
Concept
Values
Figure (a) MMATTN combines cross and self attention opera- tions between the prompt and image tokens. (b) Our CONCEPTAT- TENTION allows the concept tokens to incorporate information from other concept tokens and the image tokens, but the other way around.
Meanwhile, the image patch and prompt tokens ignore the concept tokens and attend only to each other in
textual concepts in images. CONCEPTArTENTION works by creating set of contextualized concept embeddings for simple textual concepts (e.g. "cat", "sky", etc.). These
Ozi Op (aatmaxx(qupijaa Vep.
concept embeddings are sequentially updated alongside the text and image embeddings, so they match the structure that each MMATTN layer expects. However, unlike the text prompt, concept embeddings do not impact the appearance of the image. We can produce high-fidelity saliency maps by projecting image patch representations onto the concept embeddings. CONCEPTATTENTION requires no additional training and has minimal impact on model latency and mem- ory footprint. A high level depiction of our methodology is shown in Figure 2.
A diagram of these operations shown Figure 5(b).
A Concept Residual Stream The above operations create residual stream of concept embeddings distinct from the image and patch embeddings. Following the pretrained transformer's design, after the MMATTN we apply another projection matrix P and MLP, adding the result residually to c². We apply an adaptive layernorm at the end of the attention operation which outputs several values: scale Y, shift 3, and some gating values 0n and 02.. The residual stream is then updated as
4.1. Using CONCEPTATTENTION
d+ ch + +a (Poo)
(9)
The user specifies a set of r single token concepts, like "cat", "sky", etc. which are passed through T5 encoder to produce an initial embedding e for each concept. For each MMATTN layer (indexed by L) layer-normalize the input concept embeddings c² and repurpose the text prompt's projection matrices (i.e. K p: Qp, Vp), to produce set of keys, values, and queries
c++ & clt + az MLP (1+7) (c
(10)
where +- denotes assignment. The parameters from each of our modulation, projection, and MLP layers are the same as those used process the text prompt.
k. [Kpc ... 1. [ppr, 1. [Vycn....
grxd (4)
Saliency Maps in the Attention Output Space These concept embeddings can be combined with the image patch embeddings to produce saliency maps for each layer L. Specifically, we found that taking a simple dot-product sim- ilarity between the image output vectors og and concept output vectors o produces high-quality saliency maps
One-directional Attention Operation We would like to update our concept embeddings so they are compatible with subsequent layers, but also prevent them from impacting the image tokens. Let h and u. be the keys and values of
5

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Seggeentttnnnn
PuscalVOC (Single Class)
Method
Architecture
Acc f
mloU?
mAPt
Acc t
mloUt
mAP
LRP (Binder et al., 2016)
CLIP ViT
51.09
32.89
55.68
48.77
31.44
52.89
Partial-LRP (Binder et al., 2016)
CLIP VIT
76.31
57.94
84.67
71.52
51.39
84.86
Rollout (Abnar & Zuidema, 2020)
CLIP VIT
73.54
55.42
84.76
69.81
51.26
85.34
VIT Attention (Dosovitskiy al., 2021)
CLIP VIT
67.84
46.37
80.24
68.51
44.81
83.63
GradCAM (Selvaraju et al., 2020)
CLIP VIT
64.44
40.82
71.60
70.44
44.90
76.80
TextSpan (Gandelsman et al., 2024)
CLIP VIT
75.21
54.50
81.61
75.00
56.24
84.79
TransInterp (Chefer et al., 2021)
CLIP VIT
79.70
61.95
86,03
76.90
57.08
86.74
CLIPasRNN (Sun et al., 2024)
CLIP VIT
74.05
58.80
84.80
61.76
41.48
76.57
OVAM (Marcos-ae Manchón al., 2024)
Marcos-Manchónn
SDXL UNet
79.41
88.12
73.50
58.12
87.91
DINO SA (Caron al., 2021)
DINO
81.97
69.44
86.12
80.71
64.33
88.90
DINOv2 SA (Oquab et al., 2024)
DINOv2 VIT
77.39
63.12
84.19
79.65
57.61
87.26
DINOv2 Reg SA (Darcet et al., 2024)
DINOv2 Reg
72.04
56.31
80.83
77.16
56.60
86.35
iBOT SA (Zhou al., 2022)
iBOT ViT
76.34
61.73
82.04
74.96
55,80
85.26
DAAM (Tang et al., 2022)
SDXL UNet
78.47
64.56
88.79
72.76
55.95
88.34
DAAM (Tang et al., 2022)
SD2 UNet
64.52
47.62
78.01
64.28
45.01
83.04
Cross Attention
Flux DIT
74,92
59.90
87.23
80.37
54.77
89.08
Cross Attention
SD3.5 DIT
77.80
63.67
83.50
80.22
61.46
86.97
CONCEPTATTENTION
SD3.5 DIT
81.92
67.47
90.79
83,90
69.93
90.02
CONCEPTATTENTION
Flux DiT
83.07
71.04
90.45
87.85
76.45
90.19
Table 1. CONCEPTATTENTION outperforms variety of Diffusion, DINO, and CLIP ViT interpretability methods on ImageNet-
Segmentation and PascalVOC (Single Class).
Method
Acct
mloU
( 0.) = softmax(Ox00)
(11)
TextSpan
73.84
38.10
DAAM
62.89
10.97
This is in contrast to cross attention maps which are between the image keys k and prompt queries Op-
Flux Cross Attention
79.52
27.04
CONCEPTATTENTION 86.99
51.39
We can aggregate the information from multiple layers by averaging them [ET ΣNe @o ob) where [L] denotes the number of MMATTN layers (Flux has [L] 18). These attention output space maps are unique MM-DiT models they leverage concept embeddings corresponding tex- tual concepts which fundamentally can not be produced by UNet-based models.
Table 2. CONCEPTAITENTION outperforms alternative meth- ods images with multiple classes from PascalVOC. Notably, the margin between CONCePTATTENTIOn and other methods even higher this task than when single class in each image.
5. Experiments
5.1. Implementation Details
4.2. Limitations of Raw Cross Attention Maps
Flux DIT For most of our experiments we use the Flux DiT architecture implemented in PyTorch (Paszke al., 2019). In particular, use the distilled Flux-Schnel model. We encode real images with the DiT by first mapping them to the VAE latent space and then adding varying degrees of Gaussian noise before passing them through the Flux DiT. We then cache all the concept output o, and image output vectors og from each MMATTN layer. At the end of generation we then construct concept saliency maps for each layer and average them over all layers of interest. In our experiments we leverage the activations from the last 10 of the 18 MMATTN layers.
For multi-modal DiT architectures, we could additionally consider using the raw cross attention maps
Φ(kz, )p) = softmax(q,,k]
(12)
to produce saliency maps. However, these have a key lim- itation in that their vocabulary is limited to the tokens in user's prompt. Unlike UNet-based models, multi-modal DiTs sequentially update set of prompt embeddings with each MMATTN layer. This makes it difficult to produce cross attention maps for open-set of concepts, you would need to add the concept to the prompt sequence which would then change the appearance of the image. CONCEP- TATTENTION overcomes this key limitation, and makes the additional discovery that the output space of attention mechanisms produces high-fidelity saliency maps.
Stable Diffusion 3.5 Turbo We found that our approach replicated on the Stable Diffusion 3.5 Turbo (Esser et al., 2024) DiT architecture (Figure 6).
6

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
mountain
sky
cloud
sun
"a mountain in the distance."
free
dog
sky
car
"a dog sitting on a car."
Figure CONCEPTATTENTION capable generating high quality saliency maps with Stable Diffusson 3.5 Turbo. Furthermore, the top example highlights potential failure case of CONCEPTATTENTION, The concepts "sky", "mountain", and "sun" all semantically overlap, resulting unclear object boundaries.
Space
Softmax
Acct
mloUt
mAP1
CA
SA
Acct
mloU1
mAPt
CA
66.59
49.91
73.17
52.63
35.72
70.21
CA
74.92
$9.90
87.23
51.68
34.85
69.36
Value
45.93
29.81
65.79
76.51
61.96
86.73
Value
45.78
29.68
39.61
83.07
71.04
90.45
Output
78.75
64.95
88.39
Output
83.07
71.04
90.45
Table 4. CONCEPTArTENTION performs best when we utilize both eross and self attention. We tested the effectiveness of performing just cross attention operation between the concepts and image tokens, just a self attention among the concepts, both cross and self attention, and neither. We found that doing both operations leads to
Table The output space of DIT attention layers produces more transferable representations than cross attentions. We explore the transfcrability of several representation spaces of a DiT: the cross attentions (CA), the value space, and the output space. We performed linear projections the image patches and concept vectors in each of these spaces and evaluated their performance on ImageNee--Sgmmenttion
results. Metries are computed on the ImageNet Segmentation benchmark.
generate single saliency map that are unable to generate class-specific segmentation maps. For the second dataset we leverage PascalVOC 2012 benchmark (Everingham et al., 2015). We investigate both a single class (930 images) and multi-class split (1,449 images) of this dataset. Many meth- ods (e.g. DINO) do not condition their saliency map on class, so for these methods we restrict our evaluation to examples only containing a single class and the backgrond. For methods that can accept text as conditioning we evaluate on the full dataset.
VodVideo X CONCEPTATTENTION generalizes the CogVideoX (Yang et al., 2025) multi-modal DiT video gen- eration model. The only change make is additionally averaging over the added frame dimension.
5.2. Zero-shot Image Segmentation
We interested in investigating (1) the efficacy of CoN- CEPTATTENTION to generate highly localized and seman- tically meaningful saliency maps, and (2) understand the transferability of multi-modal DiT representations to impor- tant downstream vision tasks. Zero-shot image segmenta- tion a natural choice for achieving these goals.
Key Baseline Methods We compare our approach to a variety of zero-shot interpretability methods which lever- age several different multi-modal foundation models. We compare to numerous interpretability methods compati- ble with CLIP: Layerwise Relevance Propagation (LRP) (Binder al., 2016), LRP on just the final-layer of ViT (Partial-LRP), Attention Rollout (Rollout) (Abnar & Zuidema, 2020). Raw Vision Transformer Attention (ViT Attention) (Dosovitskiy et al., 2021), GradCAM (Selvaraju et al., 2019), TextSpan (Gandelsman et al., 2024), CLIP as RNN (Sun et al., 2024), and the Transformer Attribution method from (Chefer et al., 2021) (TransInterp). We also
Datasets We leverage two key datasets zero-shot image segmentation datasets. First, we use a commonly used (Gan- delsman et al., 2024; Chefer et al., 2021) pero-shot segmenta- tion benchmark called mmageNet-Seementaaiin (Guillaumin et al., 2014). It is composed of 4,276 images from 445 categories. Each image primarily depicts a single central object or concept, which makes a good method for com- paring CONCEPTATTENTION to variety of methods which

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Layer Segmentation Performance
Diffusion Timestep Segmentation Performance
0.88
0.9
Metric Value
0.7
Metric
0.64
0.6
0.56
Pixelwise Accuracy mioU
Average Precision
0.40
Accuracy
Combined Accuracy Combined mioU
0.4
Combined Accuracy Combined Combined mAP
miaU
0.32
mAP
Combined MAP
0.3
0.24
14
16
100
200
300
400
500
6D0
700
800
900
Layer
All Noise
Timestep
No Noise
Figure 7. (Left) Later MMATTN layers encode richer features for zero-shot segmentation. We investigated the impact of using features from various MMATTN layers and found that deeper layers lead better performance on segmentation metrics like pixelwise accuracy, mloU, and mAP. We also found that combining the information from all layers further improves performance. (Right) Optimal segmentation performance requires some noise to be present the image. We evaluated the performance CONCEPTATTENTIO by encoding samples from a variety of timesteps (determines the amount of noisc). Interestingly, found that the optimal amount of noise was not zero, but in the middle to later stages of the noise schedule.
compare to UNet-based interpretability methods that aggre- gates information from UNet cross attention layers called DAAM (Tang al., 2022) on both SDXL (Podell et al., 2023) and SD2, and OVAM (Li et al., 2023b) with SDXL. We compare to the self-attention maps of various DINO models: DINOv1 (Caron et al.. 2021), DINOv2 (Oquab et al., 2024), and DINOV2 with registers (Darcet et al., 2024). Finally, we compare to the raw cross attention maps produced by Flux and Stable Diffusion 3.5 Turbo.
across all three metries (Table 1). This is true for diffusion, CLIP, and DINO based interpretability methods.
Qualitative Evaluation We show qualitative results com- paring the segmentation performance to each baseline in Figure 4 and more qualitative results in Appendix B. is worth noting that the qualitative segmentation results high- light (a) the ambiguity of zero-shot image segmentation, and (b) the limitations of human data annotation. For example, Figure 4 shows our method does not segment the part of the dog between the ears and it's body, while the ground truth does.
Single Object Image Segmentation For our first task we closely follow the established evaluation framework from (Gandelsman et al., 2024) and (Chefer et al., 2021). We per- form this evaluation setup on both ImageNet--Segmenttio and a subset of 930 VoS images containing only a single class. For each method we assume the class present in the image is known and use simplified descriptions of each ImageNet class (e.g. "Maltese dog" - "dog) this al- lows the concepts to be captured by a single token. We construct concept vocabulary for each image composed of this target class and set of fixed background concepts for all examples (e.g. "background", "grass", "sky").
Multi Object Image Segmentation We also wanted to evaluate the capabilities of our method at differentiating be- tween multiple classes in an image. However, only a subset of methods produce distinct saliency maps for open ended classes. For this experiment we compare to DAAM using a SDXL backbone, TextSpan using a CLIP backbone, and the raw cross attentions of Flux. Instead of binarizing the image to produce segmentations, for each patch we predict the concept with the highest score. We used pixelwise accu- racy and mloU as evaluation metrics and found that method significantly outperformed the baselines (Table 2). We also show qualitative results of our approach differentiaa- ing between multiple concepts in a single image in Figures 1, and we show more results in Appendix B.
Quantitative Evaluation Each method produces set of scalar raw scores for each image patch which we then thresh- old based on the mean value to produce a binary segmenta- tion prediction. We compare each method using standard segmentation evaluation metrics, namely: mean Intersection over Union (mloU), patch/pixelwise accuracy (Acc), and mean Average Precision (mAP). Accuracy alone is an insuf- ficient metric as our dataset is highly imbalanced, mloU is significantly better, and mAP captures threshold agnostic segmentation capability. We found that CONCEPTATTEN- TION significantly out performs all of the baselines we tested
5.3. Ablation Studies
We perform several ablation studies to investigate the impact of various architectural choices and hyperparameters on the performance of CONCEPTATTENTIOS
8

---

ConceptAttentiin: Diffusion Transformers Learn Highly Interpretable Features
Saliency Maps for "dog"
CONCEPT ATTENTION
Cross Attention
Time
Figure CONCEPTATTENTION generalizes seamlessly video generation MMDIT models like CogVideoX. We apply CONCEP- TATTENTION to CogVideoX (Yang et al., 2025) video generation model. We take several frames from the video and compare the saliency maps generated by CONCEPIATTENTION to the model's internal cross attention maps.
Impact of Layer Depth on Segmentation We hypothe- sized that deeper MMATTN layers in the DiT would have more refined representations that better transfer segmen- tation. This was confirmed by evaluation (Figure 7). We pull features from each diffusion layer and evaluated the segmentation performance on mmageNet-Seemmenation We also compare the performance of combining all layers simultaneously, which we found performs better than any individual layer.
5.4. Video Model Results
We include qualitative results demonstrating the efficacy of CoNcePTAiTENTION on the CogVideoX video generation multi-modal DiT model (Figure 8). Also see Figures 16 and 17 in the Appendix.
5.5. Limitations
The primary limitation of CONCEPTAITENTION is that struggles differentiate between very similar textual con- cepts. For example, for photo with sky with the sun in it, the model does not necessarily know where the bound- ary of the sun resides, instead capturing halo around the sun (Figure 6). Additionally, when no relevant concept is present, CONCEPTATIENTION will select the most similar one even if incorrect (Figure 15 in the Appendix).
Impact of Diffusion Timestep Segmentation We add Gaussian noise encoded images before passing them to the DiTs, this conforms with the expectations of the models. Intuitively one might expect the later timesteps (less noise) to have much higher segmentation performance as less in- formation about the original image is corrupted. However, we found that the middle diffusion timesteps best (Figure timestep
7). Throughout the rest of experiments 500 out of 1000 following this result.
6. Conclusion
We introduce CONCEPTARTENTION a method for interpret- ing the rich features of multi-modal DiT'. Our approach allows a user to produce high quality saliency maps of an open-set of textual concepts that shed light on how dif- fusion model "sees" an image. We perform an extensive evaluation of the saliency maps zero-shot segmentation and find that they significantly outperform variety of other zero-shot interpretability methods. Our results suggest the potential for DiT models to act as powerful and interpretable image encoders with representations that are transferable zero-shot to tasks like image segmentation.
Concept Attention Operation Ablations We compared the performance on the ImageNet Segmentation benchmark of performing (a) just cross attention from the image patches to concept vectors, (b) just self attention, (c) no attention operations, and (d) both cross and self attention. Our results seen Table 4 indicate that using combination of both cross and self attention achieves the best performance. We also investigated the impact of applying a pixelwise softmax operation over our predicted segmentation coefficients. We found that it slightly improves segmentation performance in the attention output space and significantly improves performance for the cross attention maps (Table 3)

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Impact Statement
Chefer, H., Gur, S., and Wolf, L. Transformer In- terpretability Beyond Attention Visualization, April 2021. URL http:/ /arxivivvv.. orgoorgggao wwwg/rrg/////ooooo abs/2012.
Generative models for images have numerous ethical con- cerss: they have the potential spread misinformation through realistic fake images (i.e. deepfakes), they may disrupt different creative industries, and have the potential to reinforce existing social biases present in their training data. Our work directly serves to improve the transparency of these models, and we believe our work could be used to understand the biases present in models.
09838. arXiv:2012.09838 [cs].
Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L.,, and Cohen-Or, Attend-and- =itend-and-Excite: Attention-Bart Semantic Guidance for fext-to-Image Diffusion Models. ACM Transactions Graphics, 42(4):148:11148; 10, July 2023. ISSN 0730- 0301. doi: 10.1145/3392216. URL http::: //d1 acm. org/ doi 11111...111511111111111111
Acknowledgments
Chen, M., Laina, I., and Vedaldi, A. Training-Free Lay- out Control with Cross-Attention Guidance, Novem- ber 2023. URL http:// arriv. wwt//bgg//////// abs/2304. 03373. arXiv:2304.0333 [es].
This paper is supported by the National Science Foundation Graduate Research Fellowship. This work was also sup- ported in part by Cisco, NSF #2403297, gifts from Google, Amazon, Meta, NVIDIA, Avast, Fiddler Labs, Bosch.
Cho, J. H., Mall, U., Bala, K., and Hariharan, B. PICIE: Un- supervised Semantic Segmentation using Invariance and Equivariance in Clustering, March 2021. URL http: a arxiv. aadivoorg/bba 21 2103..11 1 17700. arXiv:2103.17070 [cs].
References
Abnar, S. and Zuidema, W. Quantifying Attention Flow in Transformers, May 2020. URL http: /arxiv.orgg////// abs/2005.000 00928, Xix::2005500928 [es].
Dalva, Y., Venkatesh, K., and Yanardag, FluxSpace:
Disentangled Semantic Editing Rectified Flow Trans- formers, December 2024. URL http: / arxiv.org/ abs / 2412. 09611 1. arXiv:2412.096 [cs].
Amit, T., Shaharbany, T., Nachmani, E., and Wolf, L. SegDiff: Image Segmentation with Diffusion Probabilis- tic Models, September 2022. URL http: arxiv. brg/abs/220 2212. 00390. arXiv:2112.003: [es].
Darcet, T. Oquab, M., Mairal, J., and Bojanowski, P. Vision Transformers Need Registers, April 2024. URL
Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., and Babenko, A. Label-Efficient Semantic Segmentation with Diffusion Models, March 2022. URL http::///// arxiv. org/abs// hrrg/abs/2122. 2112. 03126. 1220v2212 [cs].
http//awwivv.gg//ee...///.e wogg/oo///////00/o 16588. arXiv:2309.16588 [cs].
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale, June 2021. URL http:// arxiv. arxiv.org/aai abs/ 2010 11929. arXiv:2010.11929 [cs].
Binder, A., Montavon, G., Bach, S., Müller, K.-R., and Samek, W. Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers, April 2016. URL http:// /ttp//aiii....abaaaa///. ort.i////g/abbb1144... 00825. arXiv:1604.00883 [es].
Brooks, T. Holynski, A., and Efros, A. A. InstructPix2Pix: Learning to Follow Image Editing Instructions, Jan- uary 2023. URL http: arraiiivooo barg/abs/22/ 09800. arXiv:2211.09800 [cs].
Epstein, D., Jabri, A., Poole, B., Efros, A. A., and Holynski, A. Diffusion Self-Guidance for Controllable Image Gen- eration, June 2023. URL http: arxiv. .ow.g/oor...... abs// 2306. C0986. arXiv:2306.009:: [cs].
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B.. Ippolito, D., and Wallace, E. Extracting Training Data from Diffusion Models, Jan- uary 2023. URL http: http:///axivv arwiv..iivv..///vv//.a////aavvvv 13188. arXiv:2301.13I1 [es].
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H. Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Good- win, A., Marek, Y., and Rombach, R. Scaling Rectified Flow Transformers for High-Resoluse
Image Synthe- sis, March 2024. URL http: /aaaiiivvvorr wtwp/v.orrraaeeeee /bb// 2403. 63206. arXiv:2403.03206.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J. Bojanowski. P., and Joulin, A. Emerging Properties Sepf-Super Vision Transformers, May 2021. URL http: arxiv. wwt.xiv.org/bb////00000 14294. arXiv:2104.1 14294 [es].
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. The Pas- cal Visual Object Classes Challenge: A Retrospective.
10

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
International Journal of Computer Vision, 111(1))98- 136, January 2015. ISSN 1573-1405. doi: 10.1007/
_eprint:
$11263-014-0733-5 URL https:///oi.oo/
Labs, B. F. FLUX, 2023 http://0w0....tttaa////hhh...b..
1007/911263-014-7333
black-forest wowest--abeeelllleeeeee
Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting CLIP's Image Representation via Text-Based Decomposi- tion, March 2024. URL http: / /arxivi...////////////a//////av/ org/ / 2310. 05916. 6. arXiv:2310.05916 [cs].
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E., and Pathak, D. Your Diffusion Model is Secretly Zero-Shot Classifier, September 2023a. URL http: / aarxiv. org/ abs abs/a303...../ 16203. arXiv:2303. [cs].
Guillaumin, M., Küttel, D., and Ferrari, V. ImageNet Auto-Annotation with Segmentation Propagation. In- ternational Journal of Computer Vision, 110(3):328- 348, December 2014. ISSN 1573-1405, doi: 10.1007/ $11263-014-0713-9, URL https: http://di...gg/g// 000-000-------11110001
Li, 7., Zhou, Q. Zhang, X., Zhang, Y., Wang, Y., and Xie, W. Open-vocabulary Object Segmentation with Diffusion Models, August 2023b. URL http: /ato:///arxvvv. arxiv.org, abs 05221. arXiv:2301.0522 [es].
Lipman, Y.. Chen, R. T. Q.. Ben-Hamu, H., Nickel, M., and Le, M. Flow Matching for Generative Modeling, Febru- ary 2023. URL http:/ /arxiv.v.... arrg/abs/2211 02747. arXiv:2210.02747 [cs].
Gupta, G., Yadav, K., Gal, Y., Batra, D., Kira, Z., Lu, C., and Rudner, T. G. J. Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Con- trol, May 2024. URL http: / att/v/xxvvv////////v//// v. org/ /rg//bs//////ar/oo abs/ 2405. 05852. arXiv:2405.05852 [cs].
Liu, B., Wang, C., Cao, T., Jia, K., and Huang, J. Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing, March 2024. URL http: / /ar wwriv.org/abs... arXiv:2403.03431 [cs].
Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N. and Freeman, W. T. Unsupervised Semantic Segmentation by Distilling Feature Correspondences, March 2022. URL http:/ http//rxvvvoo////// org/ abs/ 2203. 08414. arXiv:2203.08414 [cs].
Liu, X., Gong, C., and Liu, Q. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, September 2022. URL http: / //aa/aavvvv..// org/ abs/ 03003. arXiv:2209.0300 [cs].
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D.
Poopttt Image Editing with Cross Attention Control, Au- gust 2022. URL http: /ttp://aaxvvvvv/ii wrt.://///aa/////// 01626. arXiv:2208.01622 [es].
Marcos-Manchón, P., Alcover-Couso, R., SanMiguel, J. C., and Martínez, J. M. Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models, March 2024. URL http: /arxiv. org/ abs/////////// 2403. 42 91. arXiv:2403. 14291 [es].
Kadkhodaie, Z., Guth, F., Simoncelli, E. P., and Mal- lat, S. Generalization in diffusion models arises from geometry-adaptive harmonic representations, April 2024. URL http: / arxiv. org/org/////....oooo orggabs//310. 02557. arXiv:2310.02557 [es].
Meral, T. H. S., Simsar, E., Tombari, F., and Ya- nardag, P. CONFORM: Contrast is All You Need for High-Fidelity Text-to-lmage Diffusion Models. pp. 9005-9014, 2024. URL httpss:
Karazija, L.., Laina, I., Vedaldi, A., and Rupprecht, C. Diffu- sion Models for Open- Vocabulary Segmentation, Septem- ber 2024. URL http: arxiv. org/ hbsg/abs//20 abs/236.. 09316. arXiv:2306.09316 [cs].
thecvf atteev/..coo//oooeeeeet///222//2
Veraal_conFFPMLLLonnttttttLLLLLL Need D.ffussion_cCVRR_222___222 html.
Image_
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S.. Berg, A. C., Lo, W.-Y., Dollár, P., and Girshick, R. Segment Anything, April 2023. URL http: arxiv. org/ ab/////s//30 02643. MrXiv:2304.02644 [cs].
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, Y., Li, S.W Misra, I., Rabbat, M., Sharma, V. Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski. P. DINOv2: Learn- ing Robust Visual Features without Supervision, Febru- ary 2024. URL http:// //rp/vaxiiv////////////a///aaaa/ org/ abs/ 07193. arXiv:2304.07193 [es].
Kuhn, H. W. The Hungarian method for the as- signment problem. Naval Research Logistics Quarterly, 2(1-2):83-97 1955. ISSN 1931- URL
9193. doi: 00.1002/na.3800020111 https://ont inll ibrary. wi wiley wiley.com/dol.. com/doi/

---

ConceptAttention: Diffusson Transformers Learn Highly Interpretable Features
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.,, Bai, J., and Chintala, S. PyTTorch: An Imperative Style, High-PPrformance Deep Learning Library, Decem- ber 2019. URL http:/ / arxiv. 01703. 1121v:191..01703 [cs].
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R.,
Parikh, D., and Batra, D. Grad-CAM: Visual Explana- tions from Deep Networks via Gradient-based Localiza- tion. International Journal of Computer Vision, 128(2):
336-359, February 2020. ISSN 0920-5691, 1573-1405. doi: 10.1007//1 02391. arXiv:1610.02391
URL http://
[cs].
Peebles, W. and Xie, S. Scalable Diffusion Models with Transformers, March 2023. URL attp:// /tap::///rxvv...// /arxiv.. org/abs/ 2212. 09748. arXiv:2212.09748.
Sun, S., Li, R., Torr, P., Gu, X., and Li, S. CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor, May 2024. URL bttp:://axivv arwii..xivvo............. abs / 2312. 07661. arXiv:2312.07661 [es].
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Improving Latent Diffusion Models for Higb-Resolution Image Synthesis, July 2023. URL http: //attp://axiv... arg/abs// 2307. 01952. arXiv:2307.01952 [cs].
Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang, G., Ku- mar, K., Stenetorp, P., Lin, J., and Ture, F. What the DAAM: Interpreting Stable Diffusion Using Cross At- tention, December 2022. URL http: / arxiv.org/ abs / 2210. 04885. arXiv:2210.04885 [es].
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Vi- sual Models From Natural Language Supervision, Febru- ary 2021. URL http: / arxiv. org http//bb//10.. 00020. arXiv:2103.00020 [es].
Tian, J., Aggarwal, L.,, Colaco, A., Kira, Z., and Gonzalez- Franco, M. Diffuse, Attend, and Segment: Unsuper- vised Zero-Shot Segmentation using Stable Diffusion, April 2024. URL http: aaaiiiiiiiv....i. org/ orgg/bs/2308.. 12469. arXiv:2308 12469 [cs].
Wang. P., Zhang. H., Zhang, Z., Chen, S., Ma, Y., and Qu, Q. Diffusion Models Learn Low- Dimensional Distributions via Subspace Clustering, De- cember 2024. URL hitp: aaraiiiiivvv///../ httpviiorr//bba.. 2409. 02426. arXiv:2409.02426 [es].
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, September 2023. URL http:// /arxivv. htg/abs/19100 10683. 1rXiv:1910010660 [es].
Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. Under- standing and Improving Layer Normalization, Novem- ber 2019. URL http: arxiv. org/ abs/ 07013. arXiv:1911.07013 [es].
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L.., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C. SAM 2: Segment Anything in Images and Videos, Oc- tober 2024. URL http:/ / arxiv www. hpgg/bbs/4400 2408. 00714. 4. arXiv:2408.00714 [cs].
Yang. Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu. J. Yang, Y., Hong, W.. Zhang, X., Feng, G., Yin, D., Zhang, Y., Wang, W., Cheng, Y. Xu, B., Gu, X., Dong, Y., and Tang, J. CogVideoX: Text- to- Video Diffusion Models with An Expert Trans- former, March 2025. URL http://aras /ttp//rxxvv.//////// abs/2408 06072. arXiv:2408.06072 [cs].
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-Resolutie Image Synthesis with Latent Diffusion Models, April 2022. URL http: //a/aaxiii arxiv. org/abs/21122 10752. arXiv:2112.1073: [cs].
Zhou, J., Wei, C., Wang. H., Shen, W., Xie, C., Yuille, A., and Kong. T. iBOT: Image BERT Pre- Training with Online Tokenizer, January 2022. HtR:ttp::/ / /arxiv. org/ abs 2111. 07832. arXiv:2 21::2111.000 111078322 [cs].
Ronneberger, O.,, Fischer, P., and Brox, T. U-Net: Convo- lutional Networks for Biomedical Image Segmentation, May 2015. URL ht:t:: / /ttttt::////iiiii wrwaia wwtp:///g//aaaaeaeeeoooooosses/s 04597. 1xXiv:1505.0455
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-CAM: Visual Expla- nations from Deep Networks via Cradient-based Local- ization, December 2019. URL http: abs/1610 02391, arXiv: 1610.0291.
12

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
A. More In-depth Explanation of Concept Attention
show pseudo-code depicting the difference between a vanilla multi-modal attention mechanism and multi-modal attention mechanism with concept attention added to it.
(+) Modal Attention
(b) Multi-modal Attention with Concept Attention
def sutti nodal, attnling, txt]:
def nutti roda Lttn with concept aatnling. txt, concepti):
a conpute the keys, queries, and va values
Corpute the keys, querses, and va lues
K, in irq_v img_prooecction(in [ig]
ing_k, ing_s, isg_ -mg_projection (ing) txt.k, K, txt_q, txt_y txt_projection concept soncep Ut concept_y InL_prejeccion/eeeeeett
txt,,, txt_ txt_v txt_projection
Concat the isage and text keys, queries, vals ing txt concat(]ing Ling
Concat the image and text keys, querles, vals ing txt concatiLing, ing txt concati [ing].. txc_a]]
ki)
Kj]
ing txt_e concatt [ing_9, ext_a])
ing txt concat (ing_y, txt_vl)
ing txt_v concati (ing_t txs_v]]
Perfom attention
sequence
actention
sequence
atin_out sell_attention
ing_txi_ ing_txtyv
attn_out self_attention to ang_ixt_ ing_txt_y) Unpack Concaterate the shapeee. shape[8]1, atto aut((igghhappl00:1
Uipaack
ing sutt::ing.shape(@0 atto (0ggh
values
concept_k concati [ing_k, concept_x)) conceptt concatí
concept_x))
Perform the concept attention Lancept_att_nna vateul
ing_concept_k k.T) concept attn nap softnax (cancat atte axis=1) scale concepts natmul attn,ttt, vap, ing_concept__i
returm ing, txt
return ing, txt, concepts
Figure 9. Pseudo-code depicting the (a) multi-modal attention operation used Flux DÎT' and (b) our CONCEPTATTENTION operation. We leverage the parameters of a multi-modal attention layer to construct a set of contextualized concept embeddings. The concepts query the image tokens (cross-attention) and other concept tokens (self-altention) attention operation. The updated concept embeddings are returned in addition to the image and text embeddings.
13

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
B. More Qualitative Results
Here we show a variety of qualitative results for our method that we could not fit into the original paper.
"dog"
"ball"
"tree"
"grass"
"sky"
"background"
CONCEPT
ATTENTION
(Flux)
Input Image
Cross Attention (Flux)
DAAM (SDXL)
TextSpan (CLIP)
Figure 10. A qualitative comparison hetween our methad and several others.
"gun"
"person"
"grass"
"car"
CONCEPT ATTENTION
(Flux)
Input Image
Cross Attention (Flux)
DAAM (SDXL)
TextSpan (CLIP)
Figure 11. A qualitative comparison hetween our method and several others.
C. Concept Attention Video Generation Models
14

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
"sky"
"penguin"
"beach"
"sun"
CONCEPT ATTENTION (Flux)
Input Image
Cross Attention (Flux)
DAAM (SDXL)
TextSpan (CLIP)
Figure 12, A qualitative comparison between our method and several others.
Concept
Last Layer
Image
GradCAM
b
b
6
Ground
6
Figure 13.. A qualitative comparison between baselines ImageNet Segmentation Images. The top row shows the soft predictions of cach method and the bottom shows the binarized segmentation predictions.
Concept
Last Layer
Image
GradCAM
Figure A qualitative comparison between numerous baselines on ImageNet Segmentation Images. top row shows the predictions of each method and the bottom shows the binarized segmentation predictions.
15

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Correct concept "bike" chosen over similar concept "car" when both are given
Concepts
"car"
"street"
"background"
"bike"
"car"
"bike"
Concept Attention
"background" "street"
Closest concept "car" chosen when correct concept "bike" is not present
Concepts
"car"
"street"
"background"
"car" "background" + "street"
Concept Attention
Figure The behavior of CONCEPTÄTTENTION when multiple relevant concepts are present and when relevant one is. When multiple similar concepts are given, like "car" and "bike", the most similar one will be chosen. However, when no relevant concept is presented, CONCEPTATTENTION will fall back the relevant one, in this case "car" for the bike patches.
16

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Time
Saliency Maps for "car"
CONCEPT ATTENTION
Cross Attention
Saliency Maps for "sky"
CONCEPT ATTENTION
Cross Attention
Saliency Maps for "tree"
CONCEPT ATTENTION
Cross Attention
Saliency Maps for "dirt"
CONCEPT ATTENTION
Cross Attention
Figure 16. CONCEPTATTENTIOS generalizes seamlessly to video generation MMDIT models like CogVideoX. We apply Cox- CEPTATTENTION to Cog VideoX (Yang et al., 2025) video generation model. We take several frames from video compare the saliency maps generated by CONCEPTATTENTION to the model's internal cross attention maps.
17

---

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Time
Saliency Maps for "dog"
CONCEPT ATTENTION
Cross Attention
Saliency Maps for "sky"
CONCEPT ATTENTION
Cross Attention
Saliency Maps for "tree"
CONCEPT ATTENTION
Cross Attention
Saliency Maps for "grass"
CONCEPT ATTENTION
Cross Attention
Figure 17. CONCEPTATTENTION generalizes seamlessly to video generation MMDiT models like CogVideoX. We apply Cox- CEPTATTENTION to Cog VideoX (Yang et al., 2025) video generation model. We take several frames from video and compare the saliency maps generated by CONCEPTATTENTION to the model's internal cross attention maps.
18


Reviews:
[{"review_id": "6aKCrwUxv3", "reviewer": "Reviewer_JHK8", "summary": "The authors present a new method to extract well-refined saliency maps from pre-trained DiT models without having to perform any additional training, mainly by directly leveraging the attention weights of the multi-modal model in a clever way to establish correspondences to a set of provided ‘concepts’ that might appear in the image – providing a neat approach for improved (layer-wise) insights into these blackbox models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough response. After reading all of the reviews, we have implemented many of the requested experiments at [this anonymous website](https://concept-attention-anonymous.github.io/) and we will incorporate these updates into the camera ready paper. We are glad that the reviewers highlight the strengths of our work:\n\n1. ConceptAttention is a **simple and elegant approach** (JHK8, XwNm) that produces **high quality saliency maps** with compelling empirical results (JHK8, XwNm, Njw6)\n2. ConceptAttention **requires no additional training** (JHK8, XwNm, FGxX, Njw6)\n3. and has the potential for **practical impact** to the community (JHK8, FGxX, Njw6) and is **well written and communicated** (JHK8, XwNm, FGxX, Njw6)\n\nWe hope our responses below address your specific concerns.\n\n---\n\n> Only critical point I see is the claim that ConceptAttention has “minimal impact on model latency”\n\nWe agree this is imprecise wording. This statement indeed holds only when $c << p$ (for $c$ concepts and $p$ patches). Thankfully, for typical values of $c$ (i.e., 1, 10, 50), the patch self-attention operations $O(p^2)$ dominate the number concept attention operations $O(pc)$.  ConceptAttention on a single NVIDIA A40 for 1,5, and 50 concepts the model takes 1.12, 1.14, and 1.20 seconds respectively to perform a forward pass.\n> 1. What happens if concepts are provided as a query that are NOT contained in the image?\n\nThis is a great question! ConceptAttention is designed to pick the most relevant concept out of those given, in the same way that a zero-shot CLIP classifier would. This means if the most similar concept out of those given is incorrect then it may still be chosen.\nFor example, take an image of a bike on the street, if the concepts “car” and “background” are chosen then “car” will likely be assigned to the bike as it is more similar than \"background\". However if both “car” and “bike” are given then the correct concept “bike” will be chosen ([see Fig F](https://concept-attention-anonymous.github.io/#f)).\n> 2. How does the quality of the saliency maps change if more or fewer concepts are provided? [...] What happens if overlapping concepts are provided?\n\nConceptAttention picks the best concept for each patch out of those available. This may lead to misattribution when there are very few concepts and none match the image contents (see previous answer). However, when there are many concepts and several have similar or overlapping meanings, then ConceptAttention will still emphasize just one. This can result in one concept (i.e, \"mountain\") overpowering another valid concept like \"tree\" ([see Fig G](https://concept-attention-anonymous.github.io/#g)).\n> 3. The choice of the threshold as the mean value [is] potentially suboptimal\n\nThank you for the opportunity to clarify our choice of threshold:\n1. The mean value was chosen in an effort to strictly adhere to the evaluation protocol laid out in (Chefer et al., CVPR 2021, also used by (Gandelsman et al., ICLR 2024) which uses the mean.\n2. To prevent a particular choice of threshold favoring certain methods, we included the mean Average Precision (mAP) metric (Table 1, 2, 3) which is a *threshold agnostic metric* measuring segmentation performance that takes the weighted mean of precisions achieved across multiple thresholds.\n\n> 4. Fig 6: is the ‘combined’ information from all layers?\n\nWe collect concept and image embeddings from each of these layers, compute their projections, and then average over the layer dimension. This improves robustness to noise from individual layers.\n> 5. Potential updates to Tab 1 could be DINOv2 or iBOT.\n\nThank you for the suggestion. We implemented iBOT, DINOv2 and DINOv2 with registers. We found that our approach outperformed each of them quantitatively on the same evaluation shown in Table 1. Intriguingly, the raw self-attention maps of DINOv2 underperform compared to the DINOv1 model. An example of maps from each of these methods is [shown in Fig D](https://concept-attention-anonymous.github.io/#d).\n> 6. Do the authors have intuitions for why [the middle steps are better than late ones]?\n\nThis was an interesting result to us as well. We have observed that early steps early steps shape the semantic, high-level structure of an image (with too much noise for high quality segmentation maps) while later steps focus on high-frequency minor details. Thus, the middle steps likely offer a good balance between these two extremes.\n> 7. It would be interesting to [show] how the saliency maps progress across different layers\n\nThis is a great suggestion! See [Fig E](https://concept-attention-anonymous.github.io/#e) for these results, which align with Fig 6 from the paper.\n\n---\n\nThanks again for your feedback! If our responses and new results are satisfactory, we would greatly appreciate the reviewer increasing their score to reflect their increased confidence in our work.", "claims_and_evidence": "The main claims made during the early stages of the paper are all well substantiated through experimental evidence.   \nThe only critical point I see is the claim that ConceptAttention has “minimal impact on model latency” (l 217 right) – as this is only true for a small set of concept embeddings; As this set is included in a self-attention operation, larger sets will inevitably cause larger latencies due to the quadratic complexity of this operation!", "methods_and_evaluation_criteria": "The task of zero-shot image segmentation as main basis for the evaluation of the ‘concept maps’ is well-chosen, as it is a reasonable way to quantify the object-specific salience maps;   \n\nThe comparative baselines might be slightly skewed to the authors’ advantage, see ‘experimental designs’ section below.\n  \nThe choice of the threshold as the mean value to produce binary segmentation masks for the quantitative analysis is an understandable but potentially suboptimal choice that could distort the results – the mean is, after all, highly affected by outliers/extreme values; Choice of median, and/or top-x % as a cut-off might be more reliable (see questions);", "theoretical_claims": "No theoretical claims beyond well-known/established formulas.", "experimental_designs_or_analyses": "The comparative baselines in Fig. 5 and Table 1 might slightly be skewed to the authors’ advantage, e.g. the choice of DINO instead of the much-newer and more powerful DINOv2, or other alternatives that commonly return better saliency maps (e.g. iBOT, …);\n\nAs mentioned previously: \nIn Section 5.1, the choice of the threshold as the *mean value* to produce binary segmentation masks for the quantitative analysis is an understandable but potentially suboptimal choice that could distort the results – the mean is, after all, highly affected by outliers/extreme values;   \n$\\textrightarrow$ Choice of median, and/or top-x % as a cut-off might be more reliable (see questions); \n\nMinor: Additional ablation regarding the choice of a simple dot-product to produce saliency maps could be interesting to justify this choice (L 258 right).", "relation_to_broader_scientific_literature": "Relation to broader literature is sufficient; The authors also discuss their constraints in terms of not comparing to methods trained on large datasets like SAM.", "essential_references_not_discussed": "None that come to mind in direct relation to the work's main contributions;    \nPotential updates to Table 1 could be DinoV2 (Oquab et al., TMLR 2024) or iBOT (Zhou et al., ICLR2022)", "other_strengths_and_weaknesses": "**Strengths**:  \n*Originality & Significance:*   \n- The authors provide a simple and neat but powerful approach which yields high-quality saliency maps and allows a variety of query-concepts to be tested for, hence provides a good measure of flexibility on this axis\n- The authors’ method repurposes the already trained parameters of the underlying multi-modal DiT model, which entirely removes any need for additional training and/or fine-tuning – providing great benefit to the community\n\n*Clarity:*\n- The paper is generally well written and easy to follow, with a good number of clear visualisations (e.g. Figure 4) supporting the contributions and explanations\n\n---\n**Weaknesses**: \n\n- Missing discussion of cases where concepts are queries that are in fact NOT in the image – see questions.\n- Missing discussion and/or analysis of behaviour dependent on number of concept queries, as well as potential partial overlap of provided concepts – see questions\n- Quantitative evaluation might be skewed towards certain methods that have a clearer separation around the mean – which is the threshold the authors choose; This could be improved upon by additionally evaluating using the median or top-X% -- see questions; \n- Quality of the manuscript should be improved – there are several typos and grammatical errors that can (and should) easily be corrected (see comments)\n- Minor: Comparative methods (Table 1) are mostly pre-2022, e.g. DINO v1 instead of the more powerful v2; see questions", "other_comments_or_suggestions": "I’d suggest the authors go through their manuscript in detail and correct the typos / grammatical mistakes, e.g.\n- L 142 right: “a diffusion models” ($\\textrightarrow$ model)\n- L 173 left: “line of work attempts perform” ($\\textrightarrow$ attempts to or performs)\n- L 241 right: “at the end of attention operation” ($\\textrightarrow$ end of the attention ..)\n- L 265 right: $\\textrightarrow$ Should start upper-case after period: This is..\n- L 412 right: “pixewlise” ($\\textrightarrow$ pixelwise)\n- L 414 right: “out performed” ($\\textrightarrow$ outperformed)\n- …", "questions_for_authors": "*TLDR;* I do like the approach, as I think it is a very simple and elegant yet powerful method to provide insights! However, I'd like the authors to address a number of questions! Depending on the responses, I'm happy to update my rating!  \n\n**Major:**  \n1. What happens if concepts are provided as a query that are NOT contained in the image? I’d be curious to hear/see whether the model will be able to recognise their non-existance, or still pick out irrelevant areas as a saliency map! And are other concepts’ saliency maps negatively affected? \n2. How does the quality of the saliency maps change if more or fewer concepts are provided? Is there a ‘sweet-spot’ in terms of number of concepts? What happens if overlapping concepts are provided, e.g. “landscape” and “mountain”/”grass”?\n3. As previously mentioned, I feel like the mean as the threshold to create the binary decision / saliency map might skew the results towards methods that don’t produce outliers (which is, of course, still a valid choice). However: have the authors investigated how median as a metric, as well as smaller top-X% (e.g. top 30%) would change this? \n\n\n**Others:**   \n\n4.  Fig 6: is the ‘combined’ information from all layers, or the layers 10-18 as detailed previously in the experimental setup description? \nIndependent of this, why do the authors think the combined approach outperforms all individual layers? How exactly are the layers combined? I am slightly surprised about this result, since I’d expect e.g. the average to lie somewhere in-between the extremes; Or are the individual contributions simply making it more robust in terms of the threshold metric?   \n5.  Although not necessarily in direct competition with the proposed approach, Table 1 does list methods like DINO – however, there have been significant improvements after 2021 in terms of DinoV2 as well as other methods like iBOT, which have shown to often produce better saliency maps; I’d be good to see some results for these methods as well if possible to get a feeling how well their saliency maps perform (Note: I don’t expect your method to be better, but it would just provide more up-to-date insight to the reader!) \n\nMore of a suggestion:    \n\n6.  I feel like the “*Impact of diffusion timestep on segmentation*” section in Section 5.2 would deserve more highlighting! It is quite interesting to see that the middle diffusion timesteps perform significantly better than both early and late ones!   \nDo the authors have more intuitions why this could be?   \n7.  I think it would be interesting to the reader to include a visualisation, i.e. qualitative analysis, how the saliency maps progress across different layers throughout – in addition to the quantitative plot that’s currently shown.\n\n---\n---\n## Post-Rebuttal Update:  \nGiven that the authors have sufficiently addressed all my concerns and have provided many additional convincing insights, I am raising my score from 2 to 4 and recommend acceptance of this work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Me3wWaofZq", "reviewer": "Reviewer_XwNm", "summary": "The paper introduces ConceptAttention, a novel method for generating saliency maps based on user-defined textual concepts. These maps are of high quality and achieve state-of-the-art performance on zero-shot image segmentation benchmarks, surpassing other interpretability methods. Notably, ConceptAttention does not require any retraining and is easy to understand. This demonstrates that the features of multi-modal diffusion transformers (MMDiTs) are highly transferable and potentially beneficial for various downstream vision tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. After reading all of the reviews, we have implemented many of the requested experiments which you can see at this [anonymous website](https://concept-attention-anonymous.github.io/). We are glad to see that multiple reviewers recognize the strengths of our work:\n\n1. ConceptAttention is a **simple and elegant approach** (JHK8, XwNm) that produces **high quality saliency maps** with compelling empirical results (JHK8, XwNm, Njw6)\n2. ConceptAttention **requires no additional training** (JHK8, XwNm, FGxX, Njw6)\n3. and has the potential for **practical impact** to the community (JHK8, FGxX, Njw6) and is **well written and communicated** (JHK8, XwNm, FGxX, Njw6)\n\nThe major changes of particular relevance to reviewer XwNm are:\n\n1. **ConceptAttention works on a Video Generation model!** Watch the demo [here](https://concept-attention-anonymous.github.io/#a). \n2. **ConceptAttention generalizes to Stable Diffusion 3.5 Turbo.** Quantitative results are shown in [Fig B](https://concept-attention-anonymous.github.io/#b) and qualitative results are shown in [Fig C](https://concept-attention-anonymous.github.io/#c). \n3. **ConceptAttention is now compared against additional baselines.** \nFollowing reviewer suggestions, we implemented: DINOv2, DINOv2 with registers, iBOT, OVAM, and CLIP as RNN and found that our method outperforms each of them. [See Table B](https://concept-attention-anonymous.github.io/#b).\n\nThese updated results will all be incorporated into our camera ready paper. Below we aim to address the particular concerns of reviewer XwNm.  \n\n---\n\n> Although comparison to DINO features were explored in the paper, it would be nice to also compare to the latest version of DINO. `[...]` This work improves the DINOv2 features even further. \n\nThe reviewer raises an interesting request for comparison against DINOv2 (Oquab et al., TMLR 2024) and DINOv2 with Registers (Darcet et al., ICLR 2024), both of which are highly relevant. Sec. 5 of our original submission evaluated the performance of the self-attention maps from both of these methods, and we found that ConceptAttention outperforms both of them. \n\nOur further analysis showed that DINOv2 actually has less interpretable self-attention maps than DINOv1, despite performing better on downstream tasks. See a table summarizing these results in [Table B](https://concept-attention-anonymous.github.io/#b). Surprisingly, we also found that DINOv2 with registers under-performed compared to DINOv2. We also provided qualitative results which subjectively match the self-attention results shown in each respective paper in [Fig D](https://concept-attention-anonymous.github.io/#d). \n\n> 1. How about other MMDiT models except Flux-Schnell (e.g., Stable Diffusion 3)? Is the performance of the method equally good?\n\nYes! We followed your suggestion and implemented our approach on a Stable Diffusion 3.5 Turbo model and found that it produces competitive results on the same quantitative evaluation we conducted for Table 1 of the manuscript. ConceptAttention on SD3.5 Turbo beats all tested baselines on both ImageNet-Segmentation and PascalVOC, though ConceptAttention on the Flux-Schnell architecture is slightly better than it on most metrics. See the [Table B]() for quantitative results and [Fig C]() for qualitative results. \n\n> 2. Will the method work with usual DiT-based models (e.g., PixArt family of models)? Is there a way to make it work with this architecture?\n\nOur approach hinges upon MMDiT models that leverage multi-modal attention layers that jointly process both text and image modalities, and thus will not work with the T2I paradigm of e.g., the PixArt family.\n\n> 3. Do you think it is possible to extend your method to [video generation models]? \n\nYes! To answer your question, we implemented ConceptAttention on CogVideoX (Yang et. al, ICLR 2025) and found that our approach seamlessly generalizes to video generation models. The only difference is that we also average information over the temporal dimension. We found that ConceptAttention produces qualitatively better results than cross attention maps from the same model. See the [video demonstration](https://concept-attention-anonymous.github.io/#a).\n\n---\n\nOnce again, we thank the reviewer for their feedback. If our responses and new results are satisfactory, we would greatly appreciate the reviewer increasing their score to reflect their increased confidence in our work.", "claims_and_evidence": "Claims are well supported by experiments.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, experimental designs and analysis are good. Ablation studies are interesting and insightful.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "1) DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024: Although comparison to DINO features were explored in the paper, it would be nice to also compare to the latest version of DINO.\n2) Vision Transformers Need Registers, ICLR 2024: This work improves the DINOv2 features even further. Again, it would be great to see the performance against this method.", "other_strengths_and_weaknesses": "Strengths:\n1) The paper is well written and easy to read.\n2) The idea is simple and effective.\n3) The claims are well supported empirically.\n4) The proposed method is interesting from the interpretability perspective and potentially useful in downstream vision tasks.\n\nWeaknesses:\n1) Some missing references and comparisons (please see above).\n2) Limited model evaluation, i.e., only Flux-Schnell model was validated so far. It would be great to see the method performance with other MMDiTs too.", "other_comments_or_suggestions": "1) In Section 4.1, in equations (4), (5), and (6), there are in total $k$ concepts which I believe should be $r$ instead as it was mentioned in the beginning of the paragraph: \"The user specifies a set of $r$ single token concepts...\".", "questions_for_authors": "1) How about other MMDiT models except Flux-Schnell (e.g., Stable Diffusion 3)? Is the performance of the method equally good?\n2) Will the method work with usual DiT-based models (e.g., PixArt family of models)? Is there a way to make it work with this architecture?\n3) I also wonder about video generation DiT models? Do you think it is possible to extend your method to them? What kind of information can be extracted from there?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SQIKOL7Acu", "reviewer": "Reviewer_FGxX", "summary": "- This paper presents ConceptAttention, a method that leverages the attention of diffusion transformers (DiTs) to generate saliency maps for localizing textual concepts in images.\n- By repurposing pre-trained  DiT's attention weights, the approach produces more accurate segmentation maps without requiring extra training.\n- The work is timely, as DiTs are widespread, yet investigations to their attention were limited; this offers a fresh perspective with both scientific and practical impact.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. After reading all of the reviews, we have implemented many of the requested experiments at [this anonymous website](https://concept-attention-anonymous.github.io/) and we will incorporate these updates into the camera ready paper. We are glad that the reviewers highlight the strengths of our work:\n\n1. ConceptAttention is a **simple and elegant approach** (JHK8, XwNm) that produces **high quality saliency maps** with compelling empirical results (JHK8, XwNm, Njw6)\n2. ConceptAttention **requires no additional training** (JHK8, XwNm, FGxX, Njw6)\n3. and has the potential for **practical impact** to the community (JHK8, FGxX, Njw6) and is **well written and communicated** (JHK8, XwNm, FGxX, Njw6)\n\nWe hope our responses below address your specific concerns.\n\n---\n\n> 'However, [existing models are] limited to the tokens in the user’s prompt.’, I don’t think this is the right claim as the concept is also the user’s prompt anyway\n\nCross attention maps are by default restricted to the tokens in the user’s prompt. However, when generating images it is often desirable to segment concepts (e.g., \"background\") not explicitly in the prompt. We somehow need to add these new concepts to the prompt without impacting the generated image's appearance. Our method allows this.\n\n> Why is disabling the feedback from concept token to image token so important?\n\nConceptAttention is a method for *interpreting the representations of MMDiT models during generation,* but we can’t use it as a tool for interpretation if our concepts change the image we are studying. Hence, we need to decouple concepts from image tokens.\n\n> [Can we just use the output vectors of prompt tokens?]\n\nYep! Our first discovery was that the output space of MMDiT attention layers encode highly interpretable features. However, these maps are restricted to the prompt vocabulary (see previous answer). Our one-way attention flow removes this restriction.\n\n> The threshold used to generate segmentation masks [...] is not reported in the methodology or experiments.\n\nWe hope to clarify this:\n1. We choose the mean value of our saliency maps as the threshold. This choice was made to strictly adhere to the evaluation protocol laid out in (Chefer et al., CVPR 2021) and used in (Gandelsman et al., ICLR 2024), which both use the mean. We will more improve our description in the final paper.\n2. We were also concerned that a particular choice of threshold could favor certain methods. Thus, we included the mean Average Precision (mAP) metric (Tab 1, 2, and 3) which is a *threshold agnostic metric* of segmentation performance that measures the weighted mean of precisions achieved across multiple thresholds.\n\n> The primary quantitative results focus on a simplified single-class setting (Tab 1), with only limited results for the multiclass setting (Table 4).\n\nOur experiments focus on a single-class setting to directly compare to the many zero-shot interpretability baselines which are only capable of generating single predictions. It would be unfair to expect methods like DINO to predict sensible maps for images with multiple classes. Of the subset of methods which can produce open vocabulary saliency maps (i.e., DAAM, TextSpan, and Cross Attention) our approach outperforms each of them.\n\n> Given the various PascalVOC versions, specifying these details is essential.\n\nThank you for your feedback. We will include extensive experimental details in the appendix. Our single-class experiments cover all 20 classes in PascalVOC, but are restricted to 930 images with only one class present in them. Our multi-class experiments cover all 20 classes and all examples in the entire dataset of 1464 images, many containing multiple classes.\n> The reported results in ConceptAttention’s Tab 4 are not comparable to Tab 2 in OVAM (Marcos-Manchón et al., CVPR 2024).\n\nTab 2 in the OVAM paper shows the evaluation of DAAM and OVAM on a synthetically generated dataset (introduced by the authors) called \"VOC-sim\", which is distinct from the VOC dataset we evaluate on. VOC-sim consists of images synthetically generated with prompts “a photograph of a {classname}” (Sec 4.1 of OVAM). This dataset is completely different from the VOC dataset we used.\n\n> It would be helpful to plot the ROC curve\n\nWhile not showing the ROC curve, the mean Average Precision (mAP) metric does capture the area under the precision-recall curve.\n\n> Baseline methods need to be compared [OVAM and CLIPasRNN]\n\nFollowing the reviewer's suggestions we implemented OVAM and CLIPasRNN as additional baselines. We found that our method outperforms both of these (see [Tab B](https://concept-attention-anonymous.github.io/#b)).\n\n---\n\nThanks again! If our responses above are satisfactory, we would greatly appreciate the reviewer increasing their score to reflect their increased confidence in our work.", "claims_and_evidence": "- l.295-297: ‘However, these have a key limitation in that their vocabulary is limited to the tokens in the user’s prompt.’, I don’t think this is the right claim, as the concept is also the user’s prompt anyway, as stated in l.246-247.\n- The claim made by the work provides a simple approach to visualising DiT attention without training and is, in general, innovative. There are sufficient visual results to support the claim. However, the quantitative evaluations are relatively weak, with unclear dataset specification and missing key related works to compare (specified in the following).", "methods_and_evaluation_criteria": "- The overall evaluation is on the right track, but some specifications need clarification:\n    - The threshold used to generate segmentation masks from saliency maps is a crucial hyperparameter that significantly impacts results. However, this parameter is not reported in the methodology or experiments.\n    - The multiclass evaluation is particularly questionable. The proposed method should naturally handle an arbitrary number of classes, yet the primary quantitative results focus on a simplified single-class setting (Table 1), with only limited results for the multiclass setting (Table 4).\n    - The setup in Table 4 is unclear. For instance, how many classes from PascalVOC are included? Given the various PascalVOC versions, specifying these details is essential. Additionally, key baseline methods, such as OVAM [1] and CLIPasRNN [2], should be compared, as both provide PascalVOC results and share baseline models with this work (e.g., DAAM).", "theoretical_claims": "- A key finding of ConceptAttention is that, in multi‐modal diffusion transformers (DiT), the prompt embeddings are dynamically updated alongside the image tokens, yet the concept tokens are designed to receive information from image tokens without feedback. This one‐way update mechanism allows the concept tokens to act as a semantic “anchor”—enabling the extraction of high-fidelity saliency maps that accurately localize textual concepts while preserving the image’s appearance. In contrast, U-Net–based diffusion models use static prompt embeddings, which simplifies visualization but lacks the flexible decoupling achieved in DiT.\n- One key question arises: **Why is disabling the feedback from concept token to image token so important?** The author argued this is one key design innovation but doesn’t explain it in theory nor provide empirical ablation to support this claim.\n- Another key question is, although the author attempts distinct concept tokens with prompt tokens, they are essentially the same (both from user-provided text and encoded with the same text encoder). The only difference is how they interact with image token (i.e. queation (9) and (10). **Therefore, my question is, what if the author just replaces the concept token with the prompt token and computes the saliency in the same way as concept attention**, i.g. $o_p = softmax(q_pk_{xp}^T)v_{xp}$?", "experimental_designs_or_analyses": "- To validate how the threshold impacts the proposed method, it would be helpful to plot the ROC curve comparing it with some key baseline methods (e.g. DAAM and Rollout CLIP).\n- Important and closely related baseline methods need to be compared, e.g. OVAM [1] and CLIPasRNN [2].", "relation_to_broader_scientific_literature": "See below the ‘Essential References Not Discussed’ section.", "essential_references_not_discussed": "- The following key literature is missing:\n    - OVAM [1] is highly relevant to the proposed method, as both share the same core architecture—using a parallel \"concept prompt\" to extract attention from a pre-trained diffusion model without training. The key difference is that [1] is implemented on a U-Net-based diffusion model, while ConceptAttention is based on DiT. Given this similarity, it is crucial to include [1] in both the related works section and the quantitative evaluation. Currently, the reported results in ConceptAttention’s Table 4 are not comparable to Table 2 in [1]. For example, DAAM achieves an mIoU of 66.2–79.7 in Table 2 of [1], whereas in Table 4 of ConceptAttention, it is only 10.97. While the specific PascalVOC subset used remains unclear, such a large discrepancy is unexpected.\n    - CLIPasRNN [2], another training-free approach, should also be included in the evaluation. Specifically, zero-shot image segmentation results can be compared against Table 1 in [2].\n\n[1] Marcos-Manchón, P., Alcover-Couso, R., SanMiguel, J.C. and Martínez, J.M., 2024. Open-vocabulary attention maps with token optimization for semantic segmentation in diffusion models. CVPR 2024.\n\n[2] Sun, S., Li, R., Torr, P., Gu, X. and Li, S., 2024. Clip as rnn: Segment countless visual concepts without training endeavor. CVPR 2024.", "other_strengths_and_weaknesses": "The proposed approach, despite its similarity to [1], addresses a timely and important problem—investigating attention in DiT. The provided visual results sufficiently support this claim. The main concern lies in unclear details and missing quantitative results. Once these are clarified, I would be happy to reconsider my score.", "other_comments_or_suggestions": "The main paper is well written, but additional details, such as the experimental setup, should be included in the appendix.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQjAMiLRaT", "reviewer": "Reviewer_Njw6", "summary": "The paper introduces a method that repurposes the attention mechanisms of multi-modal diffusion transformers (DiTs) to generate highly precise and interpretable saliency maps. Instead of relying solely on traditional cross attention, CONCEPTATTENTION leverages both cross and self attention in the output space of DiT layers to produce contextualized concept embeddings. These embeddings effectively map textual concepts (like “cat” or “sky”) onto corresponding regions in images. The method operates without additional training and is lightweight, making it a practical tool for enhancing the interpretability of diffusion models. Empirical results show that CONCEPTATTENTION achieves state-of-the-art performance in zero-shot image segmentation tasks on benchmarks such as ImageNet-Segmentation and PascalVOC, outperforming several existing approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. After reading all of the reviews, we have implemented many of the requested experiments at [this anonymous website](https://concept-attention-anonymous.github.io/) and we will incorporate these updates into the camera ready paper. We are glad that the reviewers highlight the strengths of our work:\n\n1. ConceptAttention is a **simple and elegant approach** (JHK8, XwNm) that produces **high quality saliency maps** with compelling empirical results (JHK8, XwNm, Njw6)\n2. ConceptAttention **requires no additional training** (JHK8, XwNm, FGxX, Njw6)\n3. and has the potential for **practical impact** to the community (JHK8, FGxX, Njw6) and is **well written and communicated** (JHK8, XwNm, FGxX, Njw6)\n\nWe hope our responses below address your specific concerns.\n\n---\n> a few additional references would help frame the contributions even better.\n\nWe will absolutely include your suggeted references in the related works section of the final manuscript.\n\n> it remains somewhat unclear how well the approach would generalize to tasks beyond image segmentation. A discussion of these limitations could provide a more balanced perspective\n\nWe actually found that ConceptAttention [generalizes seamlessly to video generation models](https://concept-attention-anonymous.github.io/#a)! We implemented ConceptAttention on the CogVideoX MMDiT video generation model and it generates qualitatively better saliency maps than the cross attention maps. Certainly, ConceptAttention has limitations, and we are happy to discuss more in the paper. For example, see our response to `2.` below.\n\n> 1. Could you provide further insights—either theoretical or through additional experiments—on why the attention output space yields sharper and more transferable saliency maps compared to traditional cross-attention methods?\n\nGreat question! Textual information initially flows from the prompt tokens to the image patches. However, after the initial layers the image tokens themselves will encode the rich semantic information from the prompt. Cross attention only captures the *direct* contributions of text tokens to the image patches. Our approach captures both this information and the *indirect* semantic information flowing through the other image patches.\n\n> 2. How sensitive is the method to the selection and number of concept tokens?\n\nConceptAttention is designed to pick the best concept for each patch out of those available, in much the same way that a zero-shot CLIP classifier would. This may lead to misattribution when there are very few concepts and none match the image contents. See the picture of a bike in [Fig F](https://concept-attention-anonymous.github.io/#f) for example. If the concepts “car” and “background” are chosen then “car” will be assigned to the bike as it is more similar than \"background\". However if both “car” and “bike” are given then the correct concept “bike” will be chosen.\n\nOn the other hand, when there are many concepts and several have overlapping meanings, then ConceptAttention will still pick the one it decides is \"best\". This can result in one concept (i.e. \"mountain\") overpowering another, perhaps correct, concept like \"tree\". See [Fig G](https://concept-attention-anonymous.github.io/#g) for this example.\n\n> 3. Can you comment on the generalizability of ConceptAttention beyond multi-modal DiTs and the specific segmentation tasks evaluated? Do you foresee its applicability to other architectures or downstream tasks?\n\nAs mentioned above, we found that ConceptAttention [generalizes seamlessly to video generation models](https://concept-attention-anonymous.github.io/#a). Additionally, we found it also generalizes to Stable Diffusion 3.5 Turbo, another T2I MMDiT model. We quantitatively evaluated ([see Table B](https://concept-attention-anonymous.github.io/#b)) it using the same protocol from Tab 1 in the manuscript and found it outperforms existing baselines. See [Fig C](https://concept-attention-anonymous.github.io/#c) for qualitative results.\n\n> 4. Have you considered or conducted any human-centric evaluations of the interpretability provided by the saliency maps (e.g., user studies or qualitative assessments beyond segmentation metrics)?\n\nA human-centric evaluation of our approach compared to other zero-shot interpretability methods would be a great line of future work. Of particular interest would be identifying if ConceptAttention can be used by non-experts to debug models, identifying why a model may not generate a proper image that aligns with the given prompt.\n\n> 5. Are there specific failure cases or limitations of ConceptAttention?\n\nPlease see our answer to question 2 above.\n\n---\n\nOnce again, we thank the reviewer for their feedback and we hope our responses answered your remaining questions.", "claims_and_evidence": "The experimental evidence largely backs the paper’s central claims. In particular, the authors support the claim that using the attention output space (via a combination of cross and self attention) produces sharper and more transferable saliency maps by demonstrating significant improvements in zero‐shot segmentation benchmarks (as shown in multiple tables and qualitative comparisons). The ablation studies further clarify that the combination of both attention types is crucial for the observed performance gains.\n\nHowever, a couple of points could benefit from additional evidence:\n\n• The claim that these representations are “highly interpretable” is mainly evaluated through segmentation metrics. Although improved segmentation performance is a strong indicator, a more in-depth human evaluation or analysis on other interpretability aspects could further substantiate this claim.\n\n• The broader assertion regarding the transferability of DiT representations to other downstream tasks is demonstrated only in the context of segmentation. Additional experiments on diverse tasks would help confirm the generality of this transferability.\n\nOverall, while the core experimental results are convincing, the claims about interpretability and broad transferability might be seen as slightly overreaching without further supporting evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem. The paper targets the challenge of interpreting diffusion models through precise, concept-specific saliency maps, and it does so by leveraging both cross and self attention within multi-modal DiTs. The use of standard zero-shot segmentation benchmarks like ImageNet-Segmentation and PascalVOC provides a robust framework to quantitatively and qualitatively assess the quality of these saliency maps. Additionally, the comprehensive ablation studies help confirm that the specific design choices, such as the combination of attention types, directly contribute to improved performance. This setup effectively demonstrates the method's utility for the intended application without introducing unnecessary complexity.", "theoretical_claims": "The paper does not include formal proofs for its theoretical claims. Instead, the authors provide algorithmic descriptions, equations, and intuitive justifications—such as the use of linear projections in the attention output space (e.g., Equation 13)—to support the conceptual basis of the method. The claim that this approach yields sharper and more transferable saliency maps is primarily validated through extensive empirical experiments and ablation studies, rather than through rigorous theoretical proofs. Consequently, there were no formal proofs to verify for correctness.", "experimental_designs_or_analyses": "I reviewed the experimental setups and analyses, and overall they appear sound and well-aligned with the paper’s objectives. For example:\n\n- Segmentation Benchmarks:\n  The use of standard zero-shot segmentation benchmarks (ImageNet-Segmentation and PascalVOC) to evaluate the quality of the saliency maps is appropriate. These benchmarks provide widely accepted metrics (mIoU, pixelwise accuracy, mAP) that serve as a robust proxy for assessing how well the method localizes textual concepts in images.\n\n- Ablation Studies: \n  The paper includes ablations that isolate the contributions of using just cross attention, just self attention, and their combination. This analysis clarifies that the integration of both mechanisms is key to achieving superior performance. Additionally, the study on the influence of diffusion timesteps helps understand how noise levels affect segmentation performance.\n\n- Layer-wise Analysis:  \n  The experiments also examine the impact of using features from different layers of the model. This layered analysis is useful for demonstrating that deeper layers contribute more refined representations, and that aggregating information across layers further improves the results.\n\nOne potential concern is that while segmentation performance is an effective proxy for interpretability, it does not fully capture all aspects of what makes a model’s internal representations interpretable from a human perspective. A complementary human study or alternative qualitative analysis might have provided additional validation. However, given the context and common practices in this research area, the experimental designs and analyses are both reasonable and convincing.", "relation_to_broader_scientific_literature": "The paper’s contributions build directly on and extend several strands of prior work in model interpretability, transformer architectures, and diffusion models. Specifically:\n\n• Previous research has shown that attention mechanisms in models like UNet-based diffusion models can yield useful cross attention maps for localizing textual concepts (e.g., Tang et al., 2022). This work extends that idea by demonstrating that the output space of multi-modal diffusion transformers can be repurposed—using both cross and self attention—to generate even sharper, more transferable saliency maps.\n\n• In the broader literature on transformer interpretability, methods such as GradCAM, Layer-wise Relevance Propagation, and Attention Rollout have been applied to vision transformers (including models like CLIP and DINO) to visualize and understand model decisions. The proposed CONCEPTATTENTION method builds on these insights by leveraging the rich, multi-modal representations inherent to diffusion transformers, thereby offering a new perspective on how internal representations can be made more interpretable.\n\n• The paper also connects to recent work that explores how the representations of diffusion models can be utilized for downstream tasks such as segmentation. By showing that the same representations can be interpreted through concept embeddings to achieve state-of-the-art zero-shot segmentation performance, the paper bridges the gap between generative modeling and practical image analysis.\n\nOverall, the work synthesizes ideas from transformer-based interpretability and diffusion model research, advancing the understanding of how multi-modal attention mechanisms can be manipulated to yield more precise and meaningful explanations of model behavior.", "essential_references_not_discussed": "The paper is satisfactory, but a few additional references would help frame the contributions even better. For example:\n\n• TCAV (Testing with Concept Activation Vectors by Kim et al., 2018) is a seminal work on concept-based interpretability. It shows how high-level concepts can be used to explain model decisions, which directly relates to the paper’s idea of using concept embeddings to generate saliency maps. Including a discussion of TCAV would help readers see how the current approach builds on or differs from established concept-based methods.\n\n• The critique “Attention is not Explanation” by Jain and Wallace (2019) offers important context for any work that leverages attention mechanisms for interpretability. Although the authors argue that the attention output space in DiTs yields sharper and more reliable saliency maps, contrasting their findings with the limitations highlighted in that work would provide a more nuanced perspective.\n\nIncluding these related works would better situate the paper’s contributions within the broader literature on interpretability and help clarify how its proposed method advances beyond previous approaches.", "other_strengths_and_weaknesses": "Other Strengths:\n\n- Originality: \n  The paper creatively repurposes the attention mechanisms in multi-modal diffusion transformers to generate interpretable concept embeddings without requiring additional training. This inventive combination of ideas from diffusion models and attention-based interpretability represents a fresh perspective that advances the state of the art.\n\n- Significance:  \n  By demonstrating state-of-the-art performance on zero-shot segmentation tasks, the work highlights the practical impact of its method. Its ability to produce sharp, transferable saliency maps not only deepens our understanding of DiT representations but also has potential implications for enhancing the transparency and controllability of generative models.\n\n- Clarity: \n  The paper is generally well-structured and clearly written, with detailed descriptions of the methodology, comprehensive experimental evaluations, and helpful pseudo-code that clarifies the proposed approach. The extensive ablation studies further reinforce the clarity of the experimental design and results.\n\nOther Weaknesses:\n\n- Generality:  \n  The method is demonstrated on multi-modal DiTs, and it remains somewhat unclear how well the approach would generalize to tasks beyond image segmentation. A discussion of these limitations could provide a more balanced perspective.\n\n- Theoretical Underpinning: \n  The paper could benefit from a deeper theoretical analysis of why the attention output space yields superior saliency maps compared to traditional cross-attention methods. While the empirical results are convincing, additional theoretical insights would enhance the overall robustness of the claims.\n\nOverall, the paper makes a compelling contribution with its original approach and significant empirical findings, though further exploration in the areas noted above would provide additional depth and context to its contributions.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. Could you provide further insights—either theoretical or through additional experiments—on why the attention output space yields sharper and more transferable saliency maps compared to traditional cross-attention methods?  \n\n2. How sensitive is the method to the selection and number of concept tokens? For example, how does varying the vocabulary size or the choice of specific tokens affect the segmentation performance and interpretability?  \n\n3. Can you comment on the generalizability of CONCEPTATTENTION beyond multi-modal DiTs and the specific segmentation tasks evaluated? Have you explored or do you foresee its applicability to other architectures or downstream tasks?  \n\n4. Have you considered or conducted any human-centric evaluations of the interpretability provided by the saliency maps (e.g., user studies or qualitative assessments beyond segmentation metrics)?  \n\n5. Are there specific failure cases or limitations of CONCEPTATTENTION, particularly when dealing with images containing multiple overlapping or ambiguous objects?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method for extracting well-refined saliency maps from pretrained DIT models in a training-free manner by leveraging attention weights with an additional set of concept tokens. It received four reviews, and after a successful rebuttal, the final ratings are three accepts and one weak accept.

All reviewers acknowledge the novelty and effectiveness of the method, the soundness of the evaluation design, and the clarity of the presentation. Overall, in line with the reviewers, the AC believes this work addresses a timely and important problem and is likely to attract interest from the community.
Decision: Accept (oral)