Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers
Authors: Chaehyun Kim
Heeseong Shin
Eunbeen Hong
Heeji Yoon
Anurag Arnab
Paul Hongsuck Seo
Sunghwan Hong
Seungryong Kim
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion models
Open-vocabulary semantic segmentation
Image generation
Primary area: deep_learning
TL;DR: We uncover the emergent open-vocabulary semantic segmentation capability of diffusion transformers and show that amplifying this property enhances both segmentation and image generation.

Abstract: Text-to-image diffusion models excel at translating language prompts into photorealistic images by implicitly grounding textual concepts through their cross-modal attention mechanisms. Recent multi-modal diffusion transformers extend this by introducing joint self-attention over concatenated image and text tokens, enabling richer and more scalable cross-modal alignment. However, a detailed understanding of how and where these attention maps contribute to image generation remains limited. In this paper, we introduce Seg4Diff (Segmentation for Diffusion), a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image. Through comprehensive analysis, we identify a semantic grounding expert layer, a specific MM-DiT block that consistently aligns text tokens with spatially coherent image regions, naturally producing high-quality semantic segmentation masks. We further demonstrate that applying a lightweight fine-tuning scheme with mask-annotated image data enhances the semantic grouping capabilities of these layers and thereby improves both segmentation performance and generated image fidelity. Our findings demonstrate that semantic grouping is an emergent property of diffusion transformers and can be selectively amplified to advance both segmentation and generation performance, paving the way for unified models that bridge visual perception and generation.

Seg4Diff:
Unveiling Open-Vocabulan Segmentation in Text-to-Image Diffusion Transformers
Chaehyun Kim!
Heeseong Shin'
Eunbeen Hong' Sunghwan Hong St
Heeji Yoon' Seungryong Kim 4f
Anurag Arnab
Paul Hongsuck Seo?
'KAIST AI
Korea University
*ETH Zürich Al Center, CVG, PRS
https://cvlab-kaiie github Cacaub.io//eeee /ogggg4ifff
Seg4Diff
"Three eats and one deg sitting grass."
three cats
dog
Image
Emergent semantic grounding
Mask Alignment
Pretrained DIT
MM- Block
Semontic grounding expert
three cots
dog
Generated image
MM--DIT Block
Open-vocabulary segmentation
Unsupervised
Boosting segmentation and generation
segmentation
We introduce Seg4Diff, a systematic framework designed to analyze and enhance the emergent semantic grounding capabilities of multi-modal diffusion transformer (MM-DiT) blocks in text-to-image diffusion transformers (DiTs). Here, semantic grounding expert refers to specific MM-DiT block responsible for establishing semantic alignment between text and image features.
Figure 1:
Abstract
Text-to-image diffusion models excel at translating language prompts into photore- alistic images by implicitly grounding textual concepts through their cross-modal attention mechanisms. Recent multi-modal diffusion transformers extend this by introducing joint self-attention over concatenated image and tokens, enabling richer and more scalable cross-modal alignment. However, detailed understand- ing of how and where these attention maps contribute to image generation remains limited. In this paper, we introduce Seg4Diff (Segmentation for Diffusion), a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image. Through comprehensive analysis, we identify a semantic grounding expert layer, specific MM-DIT block that consistently aligns text tokens with spatially co- herent image regions, naturally producing high-quality semantic segmentation masks. We further demonstrate that applying a lightweight fine-tuning scheme with aanstannnttted image data enhances the semantic grouping capabilities these layers and thereby improves both segmentation performance and generated image fidelity. Our findings demonstrate that semantic grouping is an emergent property of diffusion transformers and can be selectively amplified to advance both segmentation and generation performance, paving the way for unified models that bridge visual perception and generation.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
The emergence of text-to-image (T2I) diffusion models have revolutionized visual content creation by allowing users to generate photorealistic images from natural-language prompts [46] 22 52). The success of these models imply that text-to-image diffusion models are highly capable of associating visual and textual representations, as the generated images accurately reflect the given textual prompt.
In this regard, a growing body of work shows that this semantic grounding is encoded in the cross- modal attention maps of the diffusion models [20]
which have already been actively exploited for generative downstream tasks such as image editing, inpainting and personalization [48] 20 39]. Furthermore, several studies [60] 54 show that the cross-attention maps can be used to perform visual perception tasks such as open-vocabulary semantic segmentation, revealing that diffusion models can double as training- produced by the standard U-Net-based [47 46) models is often noisy and spatially fragmented [6], limiting the fidelity of the resulting masks when leveraged without heuristic refinement.
segmentation models. However, the attention map
Recent advances replace the U-Net architecture with diffusion transformers (DiTs) 42 which benefits from the strong representation power and scalability transformers for achieving state- of-the-art image generation quality. In particular, the multi-modal diffusion transformer (MM- DiT) [15] 53 33 introduces multi- modal attention, which concatenates text and image tokens and applies joint self-attention, enabling richer cross- moss- interaction. This encourages us explore the internal representations and the characteristics of the MM-DiT-based models. However, as opposed to U-Net based models, which have been extensively studied [38), DiT-based models and their characteristics remain underexplored, in which a detailed understanding of how and where these attention maps contribute to image generation remains limited.
In this paper, we first conduct an in-depth analysis of the joint attention mechanism in MM-DiT mod- els. We characterize the distribution of attention scores to discover active cross-modal interaction, and complement this with feature similarity measure and norm analysis to assess which modality- textual or visual-ess exerts greater influence the output representations. Together, these perspectives isolate a small subset of layers that consistently align textual semantics with contiguous image regions. These observations motivate us to further analyze this emergent semantic grounding capability.
Subsequently, we propose an open- open-vocabulary segmentation scheme leveraging MM-DiT, which demonstrates that the identified layer, semantic grounding expert, yields high-quality segmentation masks, confirming their inherently competitive capability for open-vocabulary semantic segmentation. Extended analysis reveals that these attention map is in fact further decomposed into attention heads, which capture distinct parts of the semantic region and finally sum up to construct a complete semantic mask. Moreover, without explicit class information, reveal that <pad> tokens decompose the image into meaningful semantic regions, acting as anchors for the unconditional generation.
Building on these findings, we introduce a lightweight fine-tuning scheme, called mask alignment for segmentation and generation (MAGNET), that explicitly strengthens semantic grouping by leveraging mask-andotated image data in selected MM-DIT layers. Our primary motivation is to reinforce the emergent segmentation capability of these layers, enabling more accurate open- vocabulary semantic masks. Interestingly, this targeted adaptation also yields modest but consistent improvement in image synthesis quality as a by-product of enhancing the cross-mode modal semantic alignment. Taken together, our stepwise analysis and selective refinement establish semantic grouping as a salient property of diffusion transformers for text conditonnt image generation oonioon---eeoooo that can be leveraged with minimal compute to advance both dense recognition and generative fidelity. These results illuminate the internal dynamics of diffusion transformers and point toward unified models that excel at both generation and perception.
In summary, our contributions are as follows:
We provide in-depth analysis and investigation of learned representations of MM-DIT within its multi-modal attention layers.
We identify critical layers in MM-DiT that are essential for preserving text-conditioned semantics throughout the generation process, which reveals strong semantic grounding.
We demonstrate zero-shot segmentation scheme to extract segmentation masks from the specific layers, and further enhance generation quality by enhancing their localization capability.

---

2 Related Work
Emergent properties of diffusion models. Diffusion models, which generate images by iteratively denoising Gaussian noise, have transformed generative modeling [22 [2). U-Net-based text-to-image
models [46] surpassed GANs [88,, while recent transformer backbones 42 further raised the bar.
In particular, multi-modal diffusion transformer (MM-DiT) [15] achieve stronger scalability and semantic alignment by fusing image and text tokens through joint attention. Beyond generation, pretrained diffusion models have been adapted to perception tasks such correspondence [67] (56). segmentation [59], tracking [37], depth estimation [29] (66 and video understanding (58). On the other hand, inspired by Trompt-to-Prome
[20). which first proposed to leverage the ross-modal attention maps of -ext-to-image diffusion models revealing that text information is delicately aggregated into image generation via the attention mechanism, a rich line of works explored attention maps in different areas, such as image editing (7) and conditional generation []. Building on this, we focus specifically on analyzing the joint attention mechanism of MM-DiT, uncovering its rich semantic structure and its potential for segmentation. This perspective not only bridges generation and perception but also highlights attention central building block for semantic reasoning in diffusion models.
Leveraging diffusion models for perception tasks. Recently, text-to-image diffusion models 46 [5 have been adapted beyond generation to tackle discriminative vision tasks with remarkable success. For segmentation, ODISE [64] integrates frozen diffusion backbone with CLIP, while adapter-based methods 668 leverage cross-attention for referring segmentation. For geometry,
lightweight fine-tuning of diffusion model for depth or normal estimation [63 17 enhances spatial detail and generalization. textureless regions, while leveraging frozen features [56] still rival supervised baselines [9] [0] (24).
correspondence, methods like DiffMatch 40 achieve robustmess to
Collectively, these works show that diffusion models encode rich priors transferable to perception. However, most approaches treat diffusion feature as static representations, leaving the role of internal attention mechanism underexplored, particularly in architectures like MM-DiT.
Diffusion models for segmentation. Diffusion models [4] 15 excel at photorealistic image synthe- sis by aligning semantic cues between images and text, which has spurred efforts to repurpose their learned representations for segmentation. For example, OVDiff (28) synthesizes class-specific support images and extracts their features to form prototypes that guide opca-vocabulary segmentation, while DiffSegmmener [60) mines self- and cross-attention maps from pretrained text-to-image diffusion models to localize arbitrary objects. Building on these insights, iSeg [54] applics an entropy-reduction step to self-attention maps and iteratively fuses them with cross-attention to progressively sharpen segmentation, and Diffseg [57] demonstrates that self-attention features alone can be converted di- rectly into masks. Meanwhile, DiffCut [13] frames unsupervised segmentation a graph-partitioning problem over diffusion features. Our work not only probes the latent representations learned by diffusion models but also leverages them to jointly enhance both image generation and segmentation.
3 Seg4Diff: Segmentation for Diffusion
3.1 Motivation and Overview
The remarkable success of diffusion models in text-guided image generation [46] 43)
is primarily attributed to their attention mechanisms [46] 20), with recent architectures like multi-modal diffusion transformer (MM-DiT) [15] further enhancing performance through joint attention over image and text modalities. Despite these advancements, the precise interactions within these multi-modal attention mechanisms remain largely underexplored. In this regard, our work aims to deeply investigate these internal interactions to elucidate the principles underlying their success, and based on these insights, we propose a simple yet effective training scheme to transfer the learned representations to the semantic segmentation task.
We begin by briefly reviewing rectified flow framework (36] and multi-modal attention mechanism in MM-DiT in Sec. 3.2) Then, in Sec. 3.3 we delve into how text and image tokens interact in attention mechanism of MM-DIT. Building on this, we identify key layers critical for image-text alignment and thereby results in faithful image generation, where we introduce zero-shot segmentation framework to leverage this semantic grounding capability. Subsequently, we propose learning strategy that jointly enhances both segmentation and generative capabilities in Sec. 3.5
3

---

3.2 Preliminaries
Rectified flow framework. While diffusion models reverse predefined process using simulated data [22] (52,, conditional flow matching (CFM) [35] trains continuous normalizing flows without simulation by framing generative modeling as regression between analytic conditional vector fields and a learnable velocity field. Rectified Flow (36] further simplifies this framework by adopting a deterministic linear interpolation between data and noise distributions. Specifically, given data sample xo Páaia and noise sample (0, T), the interpolation at time & E [0, 1] is defined as:
3tt - (1- - + le, e~ NO I).
(I)
The corresponding ground-truth velocity field along this path is constant and given by Uz (2z)) = Z0.
Let (2z)) denote the predicted velocity field parameterized by the model. The training objective minimizes the discrepancy between Uz(2,) and u, (xr), leading to the flow matching (FM) loss
LFM E2-2((0,1)) ao c>.N [/
ve(x,))l
Multi-modal attention. Building upon this formulation, transformer-based architecture known as multi-modal diffusion transformer (MM-DiT), which is exemplified by Stable Diffusion (SD3) [15], results in significant improvements in generation capability. At its core, MM-DiT employs multi- modal attention (MM-Attn) mechanism that simultaneously processes image and text modalities.
Given input image / and text T. let Ting RRbuxd, and Tiext E R.xdd be the image and text embeddings, where and wr is the height and width of image latent, and is the text token length. For each modality m E {T, 7), query, key, and value embeddings Qm. Km. and Vm. are channel-wise split to 2he K and Vi for each attention head h E {1,2, 2.. ,HH,, where H is the number of heads. These embeddings are concatenated into Q", K and V I (lom (dx) respectively:
,,,,,,,,,,.
Q* = O+1, Kh = [ KT), 4 [VA; v/1,
where de is the attention embedding dimension and [ denotes token-wise concatenation. Sub- sequently, these results are leveraged to produce multi-head attention score A , which is then multiplied with value tokens to produce attention output O², The final MM-Attn(rre Fiex) after output projection Po is then split back into image and text embeddings for downstream processing:
A = softmax ( //dd..
(4)
0 A
(5)
MM-Attt (Fing: Tixt) (o A"v").
(6)
where [ denotes channel-wise concatenation. The MM-Attn module designed to handle four distinct types of query-to-key interaction: image-to-image (I21), image-to-text (12T), text-to-image (T2I), and text-to-text (T2T), which we denote A121, Ar2r, Ar2ı and Ar2r respectively. These four pathways are illustrated in Fig. 2 a), where we particularly focus on A for analysis.
3.3 Emergent Semantic Alignment
In this section, we investigate the interactions between Timg and Tiet in DIT. To do so, we decompose and analyze attention scores and the resulting features across all heads and layers. For all the analysis, we sample 50 text prompts consisting of diverse linguistic and stylistic nature from DrawBench 49) and then generate corresponding images using the pretrained model. For elarity, we report results at timestep t 8 and 28; additional analyses over other timesteps addition, we focus on Stable Diffusion 3 architecture in our main analysis, where analysis on other MM-DiT variants such as Stable Diffusion 3.5 [53] and Flux can be found in Appx.
provided in Appx. [B.] In
Decomposing joint image-text attention. We start by visualizing and quantifying the attention scores to assess interaction strengths between image and text tokens in MM-Attn. As shown in Fig, (b) and (c). we compute layer- wise attention scores and aggregate them by interaction type, equivalent to Eq. 4 Fig. 2 (b) reveals that 12T scores are disproportionately higher than 12I, even though the 12T region in (a) roughly 40x smaller than 121, indicating that 12T dominates the overall attention budget. Conversely, Fig. (c) shows that T21 interactions are relatively smaller than T2T, suggesting that text tokens primarily self-attend to preserve semantic anchors. Based on these findings, we focus subsequent analysis 12T interactions, where further discussion can be found in Appx. B.2

---

K
Kr
Ay21
Ay2r
Layer
Ayz1
A A22
Ayzı
Ayrr
Ayzı
Ayzr
Sum of attention score
Sum of attention score
(a) Multimodal attention
(b) Image attention score across layer
(c) Text attention score across layer
Figure 2: Multi-modal attention mechanism. (a) Conceptual visualization of the attention map. (b-c) Ratios of attention assigned to image vs. text tokens. The dotted line denotes the ratio under uniform attention. Higher cross-modal proportions observed in A1r and AT21-
Layer 3
Layer?
Layer
Layer
Layer 22
Layer 13
Layer 15
Layer
'A photo of confused grizzly bear colculus class."
Query
Key
Value
Figure 3: PCA visualization of query, key and value projections. PCA results demonstrate that some layers exhibit strong positional bias, whereas some layers show clear semantic groups.
Attention feature analysis. To understand how MM-DiT's strong image-text interaction operates, we analyze feature space of MM-Attn. Feature PCA visualized Fig. highhightss sev- eral layers indicate that query and key features are semantically well aligned, whereas other layers display stronger positional biases. We further analyze the average norm the value- projected features at each layer, which is com- puted Σ ||V*||2 for image, text and tual prompt tokens, which excludes following padding tokens. The attention norm serves a proxy for the dominance of aggregated infor- mation, i.e., how strongly each key token in- fluences a query token (32). As shown in Fig. similar siayers-particllann
L2-Norm
Test tokens
Layer
Figure 4: Attention feature analysis. The L2 norm of the value projection for image and text tokens reveals that certain layers exhibit signifi- cantly stronger value magnitudes for tokens compared to image tokens.
the 9th MM-DiT block-exhibi notably high attention norms for text tokens compared to that of image tokens. This pattern suggests that text information is injected primarily into semantically aligned image-token regions at these specific layers.
Attention perturbation for image-text alignment. Building on this observation, test whether these layers causally drive image synthesis and image-text alignment. Following SEG [25], we apply Gaussian blur to the image-to-text (I2T) attention maps of selected layers and regenerate images, which is shown in Fig. 3 these layers mediate alignment, attenuating their attention should reduce semantic fidelity. Indeed, perturbing specific layers which matches with semantically well-aligned layers in our previous analysis-inde induces clear mismatches between text and content, confirming their critical role in aligning image and text.
We then convert this perturbation into lightweight guidance mechanism to both validate and exploit the effect. Rather than directly perturbing the sample, guide the standard denoising trajectory
5

---

None
Layer
Layer
Layer
Layer
1ayer
Layer
"Awhite colored sandwicch
"A horse ridiig on astrorout"
Figure 5: Effect of layer-wise 12T attention perturbation. Perturbing specific layer causes pro- nounced structural degradation, whereas other layers yields minor changes.
None
Layer 3
Layer B
Layer 12
16
Layer
"A roal Iff photogrophy of
super mario, 8k Ultra HD.
"A blue pos of tomoto with a wonden lodie."
Figure 6: Effect of perturbed 12T guidance. Guiding specific layer with perturbed 12T attention score significantly enhances image quality.
with negatively perturbed companion sample
25 Specifically.
(x1,)) Eœ(#; A12r) + s( (< As2r) é A12r))
where &o is the guided noise estimate, the standard estimate, ép is the perturbed noise estimate with Arrr, which is the perturbed version of Arr, and > 0 guidance scale. As shown in Fig. 6 this procedure substantially improves image quality while modifying only the specific layer's I2T attention regions.
Together, these findings indicate that (i) semantically well-aligned layers essential for injecting text conditioning into the image, and (ii) overall synthesis quality hinges disproportionately on these layers- consiseent with [2]. which likewise identifies particular MM-DiT layers as crucial for effective image editing.
3.4 Emergent Semantic Grouping
From the previous analysis, we identified MM-DiT layers that exhibit stronger, more critical multi- modal interactions than others and encode rich semantic structure. We now examine whether this semantic grounding manifests from segmentation perspective.
Specifically, we focus on the semantic grounding capability within the 12T attention maps. Obtaining segmentation prediction begins by encoding the input image / into a latent representation Zimg using a VAE encoder EYΛE- To ensure reliable segmentation fidelity, we use an intermediate noisy latent, generated by linearly interpolating the clean latent EVA with random noise A/(0. 1) at denoising timestep f, following the rectified flow formulation [5] 36). This technique preserves spatial structure while retaining semantie content, where the ablation timestep choice can be found in Appx. B. Meanwhile, a text prompt T is formed by concatenating the ground-trutt classnames existing the input image (e.g., "car mountain sky water") and encoded into text embeddings Thext via a text encoder Ex:
Timg t. EVAE( + (1 () Ttext (T).
(8) (9)
Both embeddings, Ting and Tiexts are fed into the MM-DIT blocks. In each layer, let A be the attention map from image tokens to specific text token for head h. We compute the mean 12T

---

EvaR
add
Ayz1
Azr
nobe
"car mountoin sky water"
Ayz1
Ayzr
Figure 7: Open-vocabulee semantic segmentation scheme in framework. We generate segmentation masks by interpreting the 12T attention scores, where the score map for each text token serves direct measure of image-text similarity to produce the final prediction.
Apr
Alr
Dird
Layer
bear
bird
Layer 12
"A brown bind and blue bear."
bear
(a) Multt-granularity semantic grouping
(b) Semantic grouping unconditional generation
Figure 9: Deeper analysis on multi-modal attention mechanism. (a) multi-granularity behavior of token-level and head-level attention, and (b) emergent semantic grouping on <pad> tokens in unconditional generation scenario.
attention map, AJ2r E Hgd by averaging 12T attention A/tr E hixxiid across all heads, which is then reshaped to construct mask logit M Rhx=rx corresponding to each text token index j. Formally, the overall process summarized by the following equations:
1 H
Σ A/TT
(10)
M Reshape(AA2m) RAxw for j I....,
(11)
Since a single classname may correspond to mul- tiple text tokens, we average their respective attention maps to produce single logit map per class. This results in a final logit tensor P E Axxxx where is the number of entire classes. The final prediction is obtained by apply- ing an argmax operation at each pixel location to assign the most likely class.
Metric
51
We visualize the qualitative results in Fig. 7 and quantitative results in Fig. 8 Detailed explana- tion regarding evaluation metries, pACC, mACC and mloU, can be found in Appx. The result shows that specific layers are responsible for se- mantic grounding, where 9th layer performs the best, which aligns with previously identified in above. We refer to layers as semantic grounding expert layers.
MACC
MACC AvE
Layer
Figure 8: Segmentation performance across
layers. Semantic grounding quality varies across MM-DIT layers, peaking in the middle blocks and specifically at the 9th layer.
Multi-granularity semantic grouping in multi-head attention. We further decompose attention map of semantic grounding expert layer into individual heads to see how each text token captures complete semantic region. Fig. 9 (a) visualizes head- and token-level attention map of for the highlighted text token, AizT and Az2r respectively. The attention heads focus distinct semantic part of the object, such as ears and legs of bear. Subsequently, token-level attention map precisely delineates the target semantics, demonstrating an inherent multi-granis granular grouping capability.
7

---

fim
Luyer
Eyye
add
noise
Lappr
"a moroon
GT masks
reshape
colored car parked side of rood
Aymask
Bipartite
matching
Mask Alignment
Rly X T1e
(a) Training pipeline
(b) Mask alignment
Figure 10: Lightweight fine-tuning pipeline via mask alignment. We introduce a simple yet effective mask alignment strategy that strengthens the I2T attention maps in the semantic grounding expert layer during additional diffusion fine-tuning with a LoRA adapter.
Unsupervised semantic grouping in <pad> tokens. This grouping behavior also emerges during unconditional generation, where the entire text sequence is set to <pad>. As shown in Fig. 9 (b), attention maps from individual <pad> tokens-desplle despite lacking explicit comaantcs--coniiseenni attend to coherent regions, with different token positions specializing in different objects or parts. This allows the model discover and segment meaningful groups, and these attention maps can thus serve as effective proposals for training-free unsupervised segmentation. To leverage this, we adapt the segmentation scheme in Fig. 7 Instead of using the attention map of a classname token, select the best-matching mask from the proposals generated by the <pad> tokens.
3.5 Boosting Segmentation and Generation
Motivated by our layer-wise analysis and the causal perturbation study in Sec. 3.3 which identified specific semantic grounding expert layer in MM-DiT as the key mediator of image-tex text alignment, we introduce a simple, lightweight LoRA fine-tuning scheme, called mask alignment for segmentation and generation (MAGNET), that directly strengthens alignment in this layer. The goal is to boost semantic grouping in its 12T attention while preserving the model's generation quality; empirically, this improves both zero-shot segmentation and image synthesis.
Specifically, we optimize two complementary losses. First, for each prediction (x(xx) we apply flo-matchhng loss LFM to supervise the diffusion process. Second, enhance the semantic grouping expressed by the expert layer's I2T attention, we add a mask loss Lmaas-
Specifically, to compute Lmssk. extract the 1. H 12T attention maps from the semantic grounding expert layer for I text tokens and heads and normalize each map to form candidate mask logits. We then perform bipartite matching between these candidates and the ground-truth masks obtain one-to-one assignments [8]. Since we do not consider classification, we match masks solely based on the mask loss. For each matched pair, the loss is computed similarly bipartite matching criterion:
Lmask Afocal Liocal + Adice Ldices
(12)
where Liocal focal loss, Laice is Dice loss, and Asocal, Adice weight their contributions.
Finally, the total objective is
Liotal LFM Amask fmaas
(13)
with Amask controlling the strength of mask alignment.
Experiments
4.1 Implementation Details
For zero-shot inference, the diffusion process is fixed timestep 8 of 28 using the flow-matching Euler discrete scheduler. Training uses 10k images from either SA-1B 60 or COCO (34), with captions generated by CLg [61) following SD3's procedure [15] Head-level attention maps are used for SA-1B 1B and token-level maps for COCO to match mask granularity. Images are processed at 1024 x 1024 resolution, and each transformer layer equipped with a LoRA module rank 16, trained using AdamW with Ir = 1 10-5 default 3 parameters, and weight decay. Training runs on two NVIDIA A6000 GPUs with per-device batch size 4 and gradient accumulation for an effective batch size of 16.

---

Madel
Ards
Inan
(OCC2
Unject
PL.39
ADE
Clb
CarCLIP Ed
CLP H0l4
91.5
52.3
-7.9
28.8
45.0
ReED DS
26.3
16.9
15.3
25.3
15.3
11.2
Masiaa D3
55.
13.4
18.3
28.3
501.
14.8
SDLS
09.
68.5
25.2
16.3
52.3
Segtbur
50&5
62.0
-911
21.2
26-5
Seçiiner
515
62.8
52.6
64.1
18.1
86.7
26.3
16.5
Sey4INI
Flax Lake
83.1
20.6
38.2
23.9
1111
52.3
52.9
36.8
41.
SIpRDAE MAGNET
SD3
5A.18
55.1
52.5
30.8
3633
556
Sep4taI
SEGGNE
COCO
023
312
382
160
Segaoat MAGNET
505
36.3
55.
20.9
26,
Table Open-vocabulary semantic segmen- tation performance. Cross-modal alignment in the I2T attention maps of the semantic grounding expert layer yields competitive re- sults, further enhanced by mask alignment.
Table 2: Unsupervised segmentation performance. Although not specifically designed for unsupervised semantic segmentation, exploiting the emergent se- mantic grouping of <pad> tokens in the I2T attention maps achieves competitive results.
(b) Unsupervised segmentation Figure 11: Qualitative results segmentation. Through our Diff4Seg, MM-DiT demonstrates strong performance in (a) -pea-locybullry semantic segmentation and (b) unsupervised segmentation. Additional results are provided in Appx. FT
(a) Open-vocabulary semantic segmentation
4.2 Seg4Diff for Segmentation
Experimental setting. We evaluate method two tasks: open-vocabulary semantic seg- mentation and unsupervised segmentation. For open-vocabulary semantic segmentation, we report mloU the validation of PascalVOC (16), COCO-Object (34), Pascal Context-59 [16,, and ADE20K [69 excluding the background class. We compare U-Net diffusion-based segmentation
We note that CLIIP-based methods process entire classnames for prediction, which is not identical to our evaluation setting. For un- supervised segmentation, we report mIoU on the validation sets PascalVOC, Pascal Context-59, COCO-Object, COCO-Stuff-27 (34), Cityscapes [12. and ADE20K. We adopt mask proposal evalu- ation protocol of DiftSeg [57], including the background class for VoS, Pascal Context-59,
methods 600 (55)
well CL.IP-based approaches 33 66
and COCO- COCO-Object in line with DiffCut [3). We compare with training- faaning-freeet free methods built on diverse
backbones, including CLIP [50 70, DINO
622 and U-Net diffusion models [57
Results. Tab. summarizes our open- vocabulary semantic segmentation results. Remarkably, using only this single layer without any refinement or postprocessing, our approach achieves competitive performance on Pascal VOC and COCO-Object dataset. Moreover, ours are robust on more complex datasets, where DiT' architecture benefits from the larger and consistent spatial resolution throughout the entire layers. This result shows that MM-DIT inherently learned fine-grained semantic grounding capability during the generation process. On the other hand, as shown in Tab. 2 our method achieves competitive performance on unsupervised segmentation. This strong outcome indicates that an emergent knowledge semantic grouping exists within the model's multi-modal attention layers, which effectively function as learnable proxies for semantic classes even when occupied by content-free free <pad> tokens.
4.3 Seg4Diff for Boosting Segmentation and Generation
Experimental setting. We evaluate the image generation capability of Seg4Diff on three benchmark datasets: Pick-a-pic (31, MS-COCO 34, and SA-1B captions generated via Cog 61). To assess text-image coherence, we utilize CLIPScore (1]] which measures the semantic alignment between the text prompt and the generated image. We report T21-CompBench++ 27 to further evaluate the fidelity and compositional quality of our generated images.

---

Method
Truining
Pick-a-PP
COCO
SA-1B
Meon
Methad
Tinining
Artribun binding
Object wistiondiige
Nam.
Comp
Baseline
27,0252
26,0338
39.3422
27,1477
Sagee
+ -MAGNTT
SA
27,0547
26. 26.2318
28 4476
25,2447
Bacing
677664
0,5544
02210
0.435
03318
0.124
0.556
03719
+ MAGNET
0000
27,9409
26.2319
28,5553
27.2760
MAGNET
SA-BB
07836
0.5679
02252
RL130
0.151
03113
0.550
0.500
Table 3: CLIPScore text-to-image gen-
Table 4: T21-Comphench++ performance. Mask
eration benchmarks. Mask alignment consistently improves alignment with text prompts across various datasets.
alignment enhances attribute binding, object relation- ships, and compositional understanding compared to the baseline.
Three catz and one dog sitcing on the grass."
"The man at bot readies to swing the pitch while the sumpire lools on.
Sasaiine
Baseline
Oun
Ours
Uhrec
Figure 12: Effects of the proposed mask alignment. Mask alignment improves structural coherence and alignment between image and text.
Baseline
Baseline
Baseline
COCO
SA18
car left of
'An elephant under sea.
dogs
"A stock of plotes. A blue plate is on the green plate
"Two cats and one dog sitting on the gross."
"Five red apples and one green apple on wooden table."
Figure 13:
Qualitative results on image generation. Mask alignment improves compositional accuracy by reducing structural errors, correcting object counts, and refining colors and textures.
Results. As shown in Tab. 3 Bnd 4 fine-tuning method improves image generation quality
the baseline, achieving higher CLIPScores and better text-image alignment. On 121-Compbench+H the COCO-trained model shows clear gains in attribute binding, numerical concepts, and complex scenes, which suggests our method effectively enhances the model's s ability to render objects with their correct attributes. However, because our training scheme do not target reasoning on object interactions, the baseline model maintains a marginal lead in rendering object relationships. Qualitative results in Fig I2 and 13 further confirm that ABoost corrects attention misalignments of the baseline, producing more accurate and plausible images. Overall, our strategy effectively enhances object-centric image generation as well as segmentation performance, as previously evidenced in Tab. [ndd]
5
Conclusion
This paper introduces Seg4Diff, systematic framework that investigates the emergent capabilities of multi-modal diffusion transformers towards open-vocabulary semantic segmentation. Through an in-depth analysis of multi-modal attention mechanism, we identify semantic grounding expert layers that are critical for aligning textual semantics with corresponding image regions. We demonstrate that attention maps from these layers can be directly used achieve competitive zero-shot segmentation performance. To further enhance this capability, we introduce lightweight fine-tuning method that strengthens the semantic grouping these expert layers. This targeted approach not only leads to meaningful improvements in segmentation quality but also enhances the model's generative fidelity. Our findings reveal that semantic alignment an intrinsic property diffusion transformers that can be selectively amplified with minimal computational cost. This work paves the way toward unified models that bridge the gap between generation and perception, delivering both high-quality image synthesis and accurate semantic understanding.
10

---

Acknowledgments and Disclosure of Funding
This research was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) IRS-2019-11190075. RS-2024- 00509279, DS-2025-11212068, RS-2023-00227552, 2025-25441888 18441838, 838, 2022-2544188 and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism (RS-2024-00333 202--0045255555, RS-2024-00333068, RSS-023-00022288 0023-00266009), and National Research Foundation of Korea (RS-2024-00346599), and the ETH Al Center through an ETH Al Center postdoctoral fellowship to Sunghwan Hong.
14-222552221479, RS-2024-0045788) RS-
RS-2022--225 02225002214479 202500221225553
the Culture, Sports,
RS-
References
[1] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed perturbed-attention attentioo guidance. In European Conference on Computer Vision, pages 1-17. Springer, 2024.
[2] Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, and Daniel Cohen-Or. Stable flow: Vital layers for training-free image editing. arXiv preprint arXiv:2411.14430 2024.
[3] Black Forest Labs. Flux. /tttps://githbb.ccmm
htw.:/hhhub.ccoomb.ccc......c... www.o//baac--ffrsstteeee forestttabsrttttr 2024.
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Sealing latent video diffusion models to large datasets. arXiv preprint arXiv:231..1517,, 2023.
[5] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 18392-18400, 2023.
[6] Lingling Cai, Kang Zhao, Hangjic Yuan, Yingya Zhang, Shiwei Zhang, and Kejie Huang. Freemask: Rethinking the importance of attention masks for zero-shot video editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1898-199 1906, 2025.
[7] Mingdeng Cao, Xintao Wang. Zhongang Qi. Ying Shan. Xiaohu Qie. and Yingiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22560-22577 2023.
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34:17864-17877 2021.
[9] Seokju Cho. Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. Advances in Neural Information Processing Systems, 34:901 2021.
[10] Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(6):7174-71 7194, 2022.
[11] Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEL/CVF Conference on Computer Vision and Pattern Recognition, pages 4113-4123, 2024.
[12] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscupes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[13] Paul Couairon, Mustafa Shukor, Jean- Emmanuel Haugeard, Matthieu Cord, and Nicolas Thome. Diffeett: Catalyzing zero-shot semantic segmentation with diffusion features and recursive normalized cut. arxiv preprint Xiv: 02842, 2024.
[14] Xiaoyi Dong. Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang. Ming Zeng Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked distiiiittllation advances contrastive language-imas pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10995 100-11000 10005, 2023.

---

[15] Patrick Esser, Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi. Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
[16] M. Everingham, L.. Van Gool, C. Williams, Winn, and A. Zisserman. The pascal visual object classes (vac) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[17] Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojje Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In European Conference on Computer Vision. pages 241-258. Springer, 2024.
[18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[19] Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen. Progressive knowledge distillation of stable diffusion xl using layer level loss. arXiv preprint arXiv:2401.02677 2024.
[20] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-O Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01626, 2022.
[21] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Bras, and Yejin Choi. CLIPScore: A reference- free evaluation metric for image captioning. In Marie-Francins Moens, Xuanjing Huang. Lucia Specia. and Scott Wen-tau Yih. editors. Proceedings of the 2021 Conference Empiricul Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6883 2020.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12550 2022.
[24] Sunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong, Sangryul Jeon, Dongbo Min, and Seungryong Kim. Neural matching fields: Implicit representation of matching fields for visual correspondence. Advances in Neural Information Processing Systems, 35:13512-13526, 2022.
[25] Susung Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. Advances in Neural Information Processing Systems, 37:66743-66772, 2024.
[26] Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Deptherafter: Generating consistent long depth sequenees for open- open-world videos, arXiv preprint arXiv:2409.00 2020909,,9 2024.
[27] Kaiyi Huang. Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. Bennhht: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation JEFE Transactions on Pattern Analysis Machine Intelligence, pages 1-17, January 5555.
[28] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for open- vocabulary segmentation. In European Conference on Computer Vision, pages 299-317, 317. Springer, 2025.
[29] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9492-9502, 2024.
[30] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tele Xiao, Spencer Whitehead, Alexander C Berg, Yan Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[31] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahhuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset user preferences for text-to-image generation, 2023.
[32] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. arXiv preprint arXiv:2004.10002 2020.
[33] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.00 04883, 2024.
12

---

[34] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part 13, pages 740-755 Springer, 2014,
[35] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02777, 2022.
[36] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
[37] Run Luo, Zikai Song, Lintao Ma, Jinlin Wei, Wei Yang. and Min Yang. Diffusiontrack: Diffusion model for multi-object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 38, pages 3991-3999, 2024.
[38] Benyuan Meng, Qianqian Xu, Zitai Wang, Xiaochun Cao, and Qingming Huang. Not all diffusion model activations have heen evaluated as discriminative features. Advances in Neural Information Processing Systems, 37:55141 111114117775 2024.
[39] Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models, 2022.
[40] Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hyoungwon Cho. Seyeon Kim, and Seungryong Kim. Diffusion model for dense matching, arXiv preprint arXiv:2305.19009 2023.
[41] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1 10741, 2021.
[42] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[43] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:200..01952 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research. 21((40))1-67, 2020.
[46] Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttion image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[47] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- asssisted intervention, pages 234-24 241. Springer, 2015.
[48] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-drive generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500- 22510, 2023.
[49] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479 734944, 2022.
[50] Gyungin Shin, Weidi Xie, and Samuel Albanic. Reco: Retrieve and co-segment for zero-shot transfer. Advances in Neural Information Processing Systems, 35:33754-33767, 2022.
[51] Heeseong Shin, Chachyun Kim, Sunghwan Hong, Seokju Cho, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Towards open- open-vocabulary semantic segmentation without semantic labels. arXiv preprint arXiv:2409. 19846, 2024.
13

---

[52] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.02502. 2020.
[53] Stability AI. Stable diffusion 3.5. httpp:///giabbb..mm/ttaabllayyyp
2024.
[54] Lin Sun, Jiale Cao. Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. iseg: An iterative refinement-baat framework for training-free segmentation. arXiv preprint arXiv:2409.03209 2024.
[55] Lin Sun, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. iseg: An iterative refinement-bed based framework for training-free free segmentation, 2024.
[56] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363-1388 2023.
[57] Junjiao Tian, Lavisha Aggarwal. Andrea Colaco. Zsolt Kira, and Mar Gonzalez-France Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3554-3563, June 2024.
[58] Pedro Vélez. Luisa F Polanía, Yi Yang, Chuhan Zhang, Rishabh Kabra, Anurag Amab, and Mehdi SM Sajjadi. From image video: An empirical study of diffusion representations. In ICCV, 2025.
[59] Chaoyang Wang. Xiangtai Li, Lu Qi, Henghui Ding, Yunhai Tong, and Ming- Hsuan Yang. Semflow: Binding semantic segmentation and image synthesis via rectified flow, Advances in Neural Information Processing Systems, 37:138981 139001, 2024.
[60] Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, and Dong Xu. Diffusion model is secretly a training-free free open vocabulary semantic segmenter, 2024.
[61] Weihan Wang. Qingsong Lv. Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogylm: Visual expert for pretrained language models. Advances in Neurul Information Procensing Systems. 37:121475-12144
2024.
[62] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3124-3134, 2023.
[63] Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. What matters when repurposing diffusion models for general dense perception tasks? arXiv preprint arXiv:2403,0m 06090, 2024.
[64] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-imas -maaiaaae diffusion models. In Proceedings of the IEEE/CVF 2023.
Conference on Computer Vision and Pattern Recognition, pages
[65] Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, Heeseong Shin, Sangbeom Lim, Honggyu An, Chaehyun Kim, Jisang Han, Donghyun Kim, et al. Visual representation alignment for multimodal large language models. arXiv preprint arXiv:2509.0797) 07979, 2025.
[66] Dengke Zhang, Fagui Liu, and Quan Tang. Correlip: Reconstructing correlations clip with off-the-sheif foundation models for open- vocabulary semantic segmentation. arXiv preprint arXiv:2411. 10006,, 2024.
[67] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 0004453334555444 2023.
[68] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 5729-5777 0739577392220 2023.
[69] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302 321, 2019.
[70] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712. Springer, 2022.
[71] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Linggiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11175-11185, 2023.
14

---

Appendix
A Further implementation detail
Model configuration
Text prompts for analysis
Attention perturbation
Segmentation evaluation metries
Further details on image quality metrics
Additional experiments
Ablation on timestep choice
Comparison on 12T and T2I attention maps
Ablation on normalization method
Segmentation performance per attention layer and head Quantitative results attention perturbation
C
Generalization other baselines
Stable Diffusion 3.5
I-ux.
Additional visualizations
Image-to-text (I2T) and text-to-image (T2I) attention map
Per-head attention maps
Statistics of attention score and norm
PCA visualization of attention features
Emergent behavior of <pad> tokens
Additional qualitative results for generation
Extended 12T attention perturbation results
Extended 12T attention perturbation guidance results Image generation results of trained models
Additional qualitative results for segmentation
Open-vocabulary semantic segmentation results
Unsupervised segmentation results
G Limitations of our method
15

---

A Further implementation detail
A.1 Model configuration
We utilize the Stable Diffusion 3 (SD3) 15 model for our main analysis. SD3 originally incorporates
three vision-language text encoders: CLIP-G/14, CLIP-L/14 44], and T5-XXL 45). Due to memory constraints, we disable the T5 encoder and use only the first 77 tokens from the two CLIP encoders. We leverage the attention score from timestep = 8 of 28, where the attention logits are semantically grouped well. Further ablation on timesteps can be found at Sec. B. We used classifier-free
guidance 233 with scale 7.5 for generation if not specified.
For training and segmentation evaluation, the input images are center-cropped if non-square and resized to a resolution of 1024 x 1024. After VAE encoding, the image is further downsampled to 64 x 64 latent. The resulting attention maps are bilinear upsampled back to the original image size for segmentation evaluation.
A.2 Text prompts for analysis
We borrowed text prompts from DrawBench 49 capability of generative models. From 200 prompts in total, we randomly sampled 50 prompts for our analysis. The full list of selected prompts is shown in Fig 14
where diverse categories are included to assess the
1. A red colored car.
blask
dag.
wlaphant behind
trank
blue colored dog.
one side and the back luga on other.
4. À zed colored basana.
28. A poar cut into seven pieces arranged is a zing.
5. A white calored sandvich.
28. Rbefrnigarator.
yellow colored giraffe.
Dininrg tablez.
green cup and blue cell phone.
31. An ineegrument used for cutting cloth, paper, andr
8. A horee riding an natrosaut.
athr thdin steroial, consansaaing of tan blades led one on Loopb of the other and fhaatned is tlu xixdglfle closed by thumb inserted tgrough rings on kthe on thei vatadleea.
9. A shark in the desert.
10. Three
bllow txo paned
fitngear
11. Onn dog on streat.
12. Tuo doga on the street.
32. A 'iiyylle on top of boat.
13. One cat. and
mitting the grass.
33.
the lnft of bus.
Three
dog aitting
15. Thrse cata tvo doga sitting the grasa.
Artophagous.
16. à =rrinnguar pink stop aign. A pink stap aign in the
36. Backlatter.
shaps of triangle.
37. X phota of confused grizely bear in calculus class. Photo sthlete cat explaining it's latest ecandal
17. illustration of stall green olephant standing
bahind large nomes.
at press confermnce to journaliata
18. à small blue book sicting on a large red book.
39. Ryper-real photo of an abandoned industrial xite
19. A stack of cubes. red cube
the sitting the middle, aithing
during
Se RD. Colouring page large cata el clinbing the wifel touer in cyberpunk tuture.
photography of super
green cube.
green cube
the botton.
20. à stack of booke. A green book 15 on the top.
sicting on red book. The red baok ix in the middle. eitting on blue book, The
42. Photo of bedroon.
Lago apace atation insida kid'a
on the botton.
21. à snall vessel propelled on water by oare, sails, or an angine.
43. A spider with a rourtache bidding an equally gentlenanly granahopper a good day during his walk to work.
22. A large plant-eating domesticated nvmal with solid tail, and to carry and pull loads.
46. bridge connecting Birope North Anerica the eye
floving
ricing, racing,
Atlantic Ocean, magnifying glam over page of 1950s bataran contc.
23. in Amorican miltintional technology corpany that facuses artificial intelligence, search engine, online adverticing, cloud computing, computer soctuare, quantun coxputing, o-comerce,, and consuter electronice.
46. A realietic photo of Poseranian dressed up like
1980e professional sreatler with neon and face paint and bright green sreetling tights with bright orange boote.
24. à large thicieskiimed sentaquatic African nemal, with tucks.
47, A sign that says 'Helllo World'. 'Diffusion".
massive ja9s
25. machine resembling buman being and
49,
Skyliie with "NeurlPs" vestten with firevorke
replicate certain human rovenenta and functions autoratically.
an the sky.
26. à grocery store refrigerstor has pint cartons of milk cartons the niddle sbelt, and
50. New Yorks Sxyline with 'Google Research Pizza Cafe?
the shelf, gallon plastic juge on the bottoz shelf.
written with fireworks the sky.
Figure 14: Selected prompts for our analysis.
A.3 Attention perturbation
To assess the importance of image-to-text (12T) attention alignment across layers, we applied a Gaussian blur along the text token dimension of the 12T attention map. We used ID Gaussian kernel with standard deviation = 9 and kernel size k = 5 to accommodate the typical text token length. The blur was applied to the attention logits after softmax, using reflective padding to preserve the total attention mass per image token.
16

---

A.4 Segmentation evaluation metrics
To assess layer-wise segmentation performance Sec. 3.4 we use three standard metries: pixel accuracy (pACC), mean accuracy (mACC), and mean Intersection-ove- Union (mloU). Each provides a progressively more rigorous evaluation based on pixel-level true positives (TP), false positives (FP), and false negatives (FN) for the number classes N. pACC reports the fraction of correctly classified pixels over entire classes, which can easily skewed by large classes like background. mACC averages per-class accuracy, TP++NN treating all classes equally yet still ignoring FP; mloU is the
TP
most comprehensive, computing intersection- cattooon-vvvuuuee over-union, TP+EP+FN
TP
TP PACC TPP+FN
mACC
1 N
Σ
TP
mloU
N Σ
TP + (14)
TP+
For unsupervised segmentation, we forward noised input image with null prompt to obtain the <pad> token attention maps. These maps then greedily merged based on KL-divergence, following the procedure [57]. The resulting mask proposals are evaluated via bipartite matching with ground-truth masks, while unmatched proposals treated as false negatives.
A.5 Further details on image quality metries.
For Pick-a-Pic, we generate 500 images for five random seeds respectively. We generated 5,000 images for MS-COCO and 1,000 images for SA-1B captions.
B
Additional experiments
B.1
Ablation on timestep choice
Fig. 15 shows segmentation performance throughout different timesteps applied to the input image. Both PascalVOC and COCO-Object demonstrates the best performance on t = of 28, where we report the segmentation performance.
PEBU
Metri
Metric
Timestep
Timestep
(a) PascalVOC
(b) COCD-Object
Figure 15: Segmentation performance across denoising timesteps.
B.2 Comparison on 12T and T2I attention maps
Fig. 16 presents attention maps for 12T and T21 directions corresponding to the highlighted keywords. The I2T maps exhibit more complete and contiguous object masks, whereas T2I tends capture only partial or attenuated regions. This discrepancy likely stems from the distinct aggregation roles: 12T attention directly updates image tokens based on textual queries, while T2I updates text tokens conditioned on visual features. Given that the diffusion process ultimately operates on image tokens synthesize outputs, it is reasonable that I2T attention aligns more strongly with semantically grounded image regions. Although this observation does not constitute formal proof, the consistency of the qualitative patterns, which is further visualized Appx. D.I across prompts supports this interpretation.
17

---

12T
22
12
"One cat one dog sitting on the grass."
cat
dog
"A panda making latte art."
panda
latte
Figure 16: Comparison 12T and T2I attention maps.
B.3 Ablation on normalization method
We conduct an ablation study to determine the optimal method for normalizing attention scores. As shown in Table 5 we evaluate the open-vocabulary semantic seg- mentation performance across four configurations: using scores before or after the softmax function, with and with- additional min-max normalization to scale the logit between O to 1. The results indicate that using the raw scores directly after the softmax function yields the best performance. Consequently, we adopt this scheme for all subsequent experiments.
softmax
min-max
VOC
Object
X 83.2
55.1
/ 85.2
53.5
X 89.0
61.8
/
88.3
57.0
Table 5: Ablation on attention score normalization methods.
B.4 Segmentation performance per attention layer and head
We present semantic segmentation results obtained by leveraging the attention scores from individual layers and heads in 17 For comprehensive evaluation, measured mloU the PASCAL VOC dataset [16]. including the background class. Our analysis reveals that the middle layers consistently exhibit superior segmentation performance compared early or late layers.
Segmentation Performence per Head
-ayor
21.0
232
11.1
26.
17.
25.4
25.
303
25.4
15.7
M6.3
I6
21.2
15.2
21.3
203
ITA
27.3
12.4
11.7
19.6
16.2
31.2
514
353
163
204
37.6
31.
135
16.
13.
X-3
14.1
15.1
IS:
F23
12:1
LE3
IRS
12:
17.7
17.5
1LK
LF-
253
L54
G/3
LC-
14.2
212
22.4
28.
20.2
22.8
25.4
2.4
22.3
22.4
21.3
2-3
22.3
22.5
23.8
22.3
36.2
22.5
315
2
203
36.3
2033
2000
615
16.
41
3
352
234
1/-
21.4
303
36.
36.
27A
16.
15.5
SF.
Laper
W
144
154
16B
App
26.
2
3LK
262
X-3
28L
31.
3
3L.
S-2
259
2.7
3L.
31
SAA
15.3
213
34.0
Lond
25.
25.
21.3
301
26.
207
2044
24-3
28.3
SK2
2012
21.
20.8
SLL
23.
2-4
23.5
210
23.3
23.2
28.3
24.5
Jow
3044
20.2
26.2
281
264
25.7
26.2
20.1
S8.
25.3
20.
26.1
25-1
26.5
25.1
21.1
21.4
213
28-
28.5
25.5
26-3
25.2
Ldpw
23
20.
26-
363
X24
3866
26.4
28L
20/4
22.4
22.8
X:4
T6-
X-3
FED
262
25-3
21
214
23.9
2-5
23
N12
205
363
213
28.9
2.3
234
344
22
3.9
21
2.5
2
153
2:3
2L-7
J6B
26-2
25.
2L-7
LF4
254
2--5
2577
2-8
24
Figure 17: mIoU score of each head.
18

---

B.5 Quantitative results on attention perturbation
We use CLIP-I, CLIP-T. and DINO scores to evaluate generation quality and alignment. CLIP-I measures cosine similarity between CLIP embeddings of generated and reference images, reflecting high-level perceptual fidelity. CLIP-T compares the CLIP embedding of a generated image with that of the conditioning text, assessing image text alignment. DINO instead computes cosine similarity between seff-supervised vision embeddings of generated and reference images, offering language- free measure of semantic similarity. Higher values indicate better fidelity or alignment, with CLIP-I and DINO focusing on image-image consisteney and CLIP-T on image-tex! text consistency.
Aligned with the qualitative analysis in Sec. 3.3 9th layer, which we designated as a semantic grounding expert in SD3, shows noticeable drop on image fidelity score when perturbed.
CLIP-/ / DINO Score
0.300
0.8
CLIPT
0.7
0.290
0.6
CLIPT
0.285
DINO
0.5
2
5
10
12
13
14
16
17
18
21
Layer
Figure 18: Image fidelity scores under layer-wise perturbations SD3.
C Generalization to other baselines
While we mainly leverage Stable Diffusion 3 (SD3) 155 our main analysis, we also apply our analysis to the other MM-DiT variants, Stable Diffusion 3.5 (SD3.5) (33] and Flux.1-dev [3]. We can similarly observe the correlation between value norm and segmentation performance among layers for both models. While SD3.5 appears to have layer 9, identical to SD3, to exhibit strong semantic grounding. Flux shows layer and 17 have similar tendency. This hints that our observation and methodology are not proprietary for SD3, but can be applied to other DiT-based diffusion models with multi modal attention, highlighting the generalizability of our insights and findings.
19

---

C.1 Stable Diffusion 3,5
Waae nowe Test Thara Thbor
25
Lyyer
Figure 19: Average L2-Norm of values across layers on SD3.5.
1.00
0.320
0.90
0.318
CUP-1 DINO Score
CUP- Score
0.85
0.316
0.314
0.70
0.312
CUP-T
0.65
DINO
10
11
12
13
14
15
16
18
19
20
21
22
23
Figure 20: Image fidelity scores under layer-wise perturbations on SD3.5.
36
misU MALL
80-
Timestep
Figure 21: Segmentation performance across layers on SD3.5.
20

---

C.2 Flux.1-dev
secrept ceee!
Figure 22: Average L2-Norm of values layers Flux.
0.3045
0.3040
0.3035
DINO Score
CLIPT Score
0.3030
0.3025
0.3020
CUPT
0.3015
DIN
Layer
Figure 23: Image fidelity scores under layer-wise perturbations on Flux. We evaluate only for the first 19 layers, which employs MM-DiT architecture.
mEU
PACC:
85
Wetres
Figure 24: Segmentation performance across layers Flux.
21

---

D Additional visualizations
D.1 Image-to-text (I2T) and text-to-image (T2I) attention map
We provide extended visualizations of image-to-text (12T) and text-to-image (T2I) attention maps Fig. 25 The maps are taken from layer 9, where we observe strong semantic alignment between visual and textual modalities. These results offer insight into the emergent cross-modal grounding dynamics of the multi-modal diffusion transformer (MM-DiT).
dog
the
street
<Jeos|
dog

>
the street
</eos|>


shark
in the desert
</eos|
"A shark in the desert."


shark
in the desert
</eos|>
<|sos|
pro pelled
by
sails OF
engine
cleos
propelled engine."
<| </sos|>
small
vessel
pro
pelled
on
water
sails
engine
eos
Figure 25: Image-to-Ter Text attention map visualization of all prompt tokens.
22

---

D.2 Per-head attention maps
We present comprehensive visualizations attention maps for individual heads in selected layers, as shown in 26g and F These results illustrate that each attention head focuses on distinct image regions. This effect is particularly pronounced layer 9, where individual heads attend to different parts of the corresponding semantic region.
"A brown bird and blue bear."
Mean
Individual Head Attention
Lower 5
Loyer'S
Loyer 12
Figure 26: Visualization attention scores for heads.
23

---

"lllustration mouse using mushroom as an umbrella."
Mean
Individual Head Attention
Layer 7
Layer 8
Layer 9
Figure 27: Visualization of attention scores for all heads.
24

---

D.3 Statistics of attention score and norm
We present the statistics of L2 norms of value-projected features across all layers and heads in Fig Notably, layer 9 exhibits consistently large norms for text tokens, suggesting that text features strongly dominate this layer. When restricting the analysis the actual prompt tokens only, this trend becomes even more pronounced.
On the other hand, observe large image token norms in specific heads within layers and 8, which coincide with the heads that show strong segmentation performance Fig. 17 This suggests that these heads are highly responsive to image-specific features and may play crucial role in localizing visual semantics prior to cross-modal alignment with text.
Figure 28: Average L2 norm of value tokens by layer and head.
D.4 PCA visualization of attention features
We provide PCA visualizations of the query-, key-, and value-projected image features across all layers in Fig. 29 Most layers exhibit a strong positional bias, whereas layer 9 reveals distinct semantic grouping. This suggests that image features become semantically well-grounded at layer 9, enabling more meaningful cross-modal interactions.
D.5 Emergent behavior of <pad> tokens
In Fig. 30 visualize all 77 tokens under the unconditional generation setting described in Sec. 3.4 which includes <sos>, <eos>, and 75 <pad> tokens. Remarkably, we observe that individual <pad> tokens attend to distinct semantic regions, despite the absence of explicit semantic information in the text prompt.
25

---

bicycle top boat"
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Query
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer 16
Layer 17
Layer 18
Layer 19
Layer 20
Layer
Layer
Layer
Layer 0
Layer 1
Layer 2
Layer 3
Layer
Layer
Layer
Layer
Key
Layer 8
Layer 9
Layer 10
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
Layer 16
Layer 17
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Layer 0
Layer 1
Layer 2
Layer
Layer
Layer
Volue
Layer 8
Layer 9
Layer 10
Layer 11
Layer 12
Layer
Layer
Layer
Layer 16
Layer 17
Layer 18
Layer
Layer
Layer
Layer
Layer
Figure 29: PCA visualization of query-, key-, and value-projected feature.
26

---

Figure 30: Emergent behavior of <pad> tokens in unconditional generation.
27

---

E Additional qualitative results for generation
We present extended results from attention perturbation experiments in Sec Sec Perturbations applied to layers other than layer 9 result in minor degradation of image fidelity or structure while largely preserving semantic content. In contrast, perturbing layer leads to the generation of semantically irrelevant images. Conversely, if we leverage this perturbed sample guidance, we observe a substantial gain on image quality. This strongly supports our claim that layer 9 plays a critical role in cross-modal interaction, with particular emphasis aligning with the text modality.
negative sample for the
E.1 Extended 12T attention perturbation results
Layer
Layer
Layer 2
Layer
Layer
Layer 5
Layer
Layer 7
"A blue colored "
Layer
Layer 10
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
Layer
Layer 18
Layer
Layer 21
Layer
23
Layer 0
Layer
Layer 2
Layer
Layer
Layer 5
Layer
Layer 7
yellow colored giraffe"
Layer
Layer
Layer 10
Layer
Layer 12
Layer 13
Layer
Layer 15
Layer
Layer 18
Layer
Layer
Layer
Layer
Layer
Layer 2
Layer
Layer
Layer 5
Layer
Layer 7
white colored sandwich." *
Layer 8
Layer
Layer 10
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
Layer
18
Layer
21
Layer
23
Figure 31: Effect of applying attention perturbation at different layers during image generation.
28

---

E.2 Extended 12T attention perturbation guidance results
Layer
2
5
Layer
Layer
"A lorge plant-eating domesticated mammal with solid hoofs flowing mane and tail, used for riding, racing, and carry and pull loads."
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer 16
Layer 17
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Layer
Layer
Layer
Layer
green cup and blue phone."
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer 16
Layer 17
Layer 18
Layer 19
Layer
Layer 21
Layer 22
Layer 23
Layer
Layer 2
Layer
5
Layer
Layer
"A car on the left of bus."
Layer
Layer
Layer 10
Layer
Layer
Layer 13
Layer
Layer
Layer 16
Layer 17
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Figure 32: Effect of applying attention perturbation guidance at different layers during image generation.
29

---

E.3
Image generation results of our trained models
Baseline
COCO
SA1B
Baseline
COCO
SA1B
"The image captures on airport tarmac during sunset. Several airplanes are porked, with one prominently displaying the "Philippine Airlines' logo. The sky is painted in hues of orange and yellow, and the ground is illuminated by the setting sun, casting warm glow over the scene."
"One car on the street." "
"A cow.
"The image depicts a playground setting with a child on a yellow slide. The child is wearing beige hoodie with the words 'PLAY AREA' printed on it, gray pants, and blue shoes. Behind the child, there's another child partially visible, wearing a red outfit. The playground has colorful elements, including a yellow giraffe silhouette and blue and red blocks."
"Five dogs on the street." "
Hello world
Hello Worlld
WAMINDS
Hello World...
"A sign that says 'Hello World'."
PICKK A PIC
PICK A PIC
PICK A PIC
"A sign that says PICK A PIC."
"The image captures a serene park setting with a long poved pathway flanked by tall, leafless trees on either side. On the right, there are stone lanterns with intricate designs, and few visitors can be seen walking along the path."
"A red book and a yellow vase."
Figure 33: Qualitative results of trained models.
30

---

Additional qualitative results for segmentation
E1
Open-vocabulary semantic segmentation results
Figure 34: Open-vocabulary semantic segmentation results of Seg4Diff on COCO-Object.
Figure 35: Open-vocabulary semantic segmentation results of Seg4Diff on Pascal-Context59.
Figure 36: Open-vocabulary semantic segmentation results Seg4Diff on ADE20K.
31

---

F2
Unsupervised segmentation results
Figure 37: Unsupervised segmentation results of Seg4Diff on COCO.
Figure 38: Unsupervised segmentation results of Seg4Diff on VOC.
Figure 39: Unsupervised segmentation results of Seg4Diff on ADE20K.
32

---

G Limitations of our method
Our method evaluates segmentation without postprocessing or upsampling, which limits performance on extremely small objects. In addition, diffusion models may ground class names differently from the ground-trum -ruth annotations, introducing an inherent mismatch that impacts accuracy. Addressing -mmamttttoonnnnnttatiinn gap is an important direction for future work.
this representation annotation
We focus on dense perception and image generation tasks, deferring reasoning-centir evaluations [65] (e.g., action recognition, spatial-relations QA) to future work, as our supervision targets segmentation semantics rather than high-level reasoning. For simplicity, the loss is applied to single "sweet-spot" layer per backbone; broader layer/timestep exploration or auxiliary heads (c.g., optical flow, pose) could yield further gains without changing our core findings.
33

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The checklist answers
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] providedd a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for dataset we used"). "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
appear either in the main paper the
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state the claims and contributions in abstract and introduction, Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are not discussed in the main paper but will be included in the supplementary material.
Guidelines:
34

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or a speech-to-tens system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification:
The paper includes mathematical formulations and algorithmic descriptions; however, it does not present any formally stated theorems or lemmas accompanied by complete set of assumptions and rigorous proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides sufficient details reproduce the main experimental results, including descriptions of the model architecture, training procedures, datasets used, and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reprodueible is important, regardless whether the code and data provided or not.
35

---

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
DDppending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have released the code and data to support further reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines
for more details.
While encourage the release of code and data, we understand that this might possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https wwt.isssionoicyyycoooooo for more details.
to
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and baselines. If only a subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended. but including URLs data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide experimenta setting/details in implementation detail section.
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide statistical value in analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: We provide information on the computer resources implementation detail section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental
well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httts::/neurrppp:: httpa//wwppp....paaaaaaccccaaaaa
Answer: [Yes]
Justification: This research fully conforms to the NeurIPS Code of Ethics. It does not involve any personally identifiable information, sensitive attributes, or human subjects, and no ethical concerns were identified in the process of conducting this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
37

---

The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work does not directly target any specific application domain. As such, the work does have any immediate or foresecable impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper uses publicly available datasets and pretrained models, all of which properly cited with their original sources.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
38

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, paperswithcode wwwode..ct....... com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset. For existing datascts that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their sub-
missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsoourcing nor research with human subjects, Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
39

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, non- standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve
LLMs any important, original, non-standard components.
Please refer to our LLM policy htttps://neurips http:///oeeeeeeeee////////////// /ttppp////errrpppc/////////raraa should or should not be described.
for what
40


Reviews:
[{"review_id": "fixOEezS74", "reviewer": "Reviewer_FeAF", "summary": "This paper systematically analyze the attention structure of MM-DiT, especially Stable Diffusion 3, and reveal its implicit capabilities in zero-shot open vocabulary semantic segmentation. By decomposing the attention score distribution and the attention norm, identified a small number of attention heads that can consistently align text tokens with spatially coherent image regions, naturally producing high-quality zero shot segmentation masks. Based on this finding, they propose a lightweight LoRA based finetuning method to enhance the semantic grouping ability of these heads without degrading the image generation quality.", "questions": "see weakness, I'm happy to raise my rating if authors can solve my concerns.", "limitations": "see weakness, mainly some experiments limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. indepth analysis on the attention mechanism, including the distrubution, L2 norms, and etc.. It is critical for identifying the property of the text to image alignment and image to text alignment.\n2. the lora based method is simple but useful, it can improve the segmentation performance while maintain the generation quality, and have the capability of zero shot on multiple dataset.\n3. This paper provide a strong theoretical support for the field of unified image generation and segmentation.\n\n## Weaknesses\n1. Only did experiments on SA-1B, in my opinion it is too detailed in some case, it would be better to provide the performance of the model trained on COCO.\n2. ODISE is a strong model for the open vocabulary segmentation method that using diffusion model, there's no quantitative comparison provided for ODISE, and some more recent work (maybe consider SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow)\n3. line 215, you mentioned that you we fix the diffusion process at half of your 28 steps and apply the classifier free guidance, do you have any ablation of the choice of step 14 and step 28?", "was_revised": true, "final_justification": "The author's rebuttal solved my concers, and they have conducted a comprehensive experiments to show the advantages of their method. Their method is well designed and well contributed, I suggest to accept.", "rebuttal": "### **Generalizability to different models**\n\nTo demonstrate that our insights are not tied to SD3, we replicated the entire analysis on two additional MM‑DiT variants—SD 3.5 [1] and Flux [2]. Summarizing the results, we observed consistent trends across all models (Note that the exact layer‑wise statistics can differ between models).\n\nBecause the rebuttal cannot include external media, we present the key quantitative results here and will supply the corresponding plots and qualitative examples in the camera‑ready version. Following the methodology in the main paper, we use three complementary evaluation protocols:\n\n- **Attention‑norm analysis** – We compute the value‑vector norms for image, text, and actual prompt tokens at every layer, highlighting only the most informative layers for readability.\n- **Attention‑perturbation analysis** – For each layer, we perturb its cross‑attention and measure DINO, CLIP‑I, and CLIP‑T similarities between the original and perturbed images; larger drops indicate layers that are critical to image fidelity.\n- **Segmentation evaluation** – Using cross‑attention maps as pseudo‑masks, we evaluate mIoU on Pascal VOC to assess mask quality.\n\nIn the table, bold indicates the highest entry for value norm and segmentation metrics, and the lowest for perturbation metrics.\n\n**Stable Diffusion 3.5 Medium**\n\n| Layer | 3 | 7 | 8 | 9 | 12 | 13 | 15 | 16 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ***[Value norm ratio]*** |  |  |  |  |  |  |  |  |\n| Image tokens | **48.5** | 39.5 | 39.1 | 27.7 | 35.8 | 32.1 | 37.3 | 42.8 |\n| Text tokens | 17.4 | 29.3 | 29.8 | **41.5** | 27.4 | 41.2 | 33.3 | 23.2 |\n| Prompt tokens | 22.9 | 35.6 | 37.3 | **53.8** | 37.0 | 47.2 | 44.8 | 32.0 |\n| ***[Perturbation metric]*** |  |  |  |  |  |  |  |  |\n| DINO | 0.8910 | 0.7814 | 0.8314 | **0.6011** | 0.8432 | 0.7929 | 0.8507 | 0.9056 |\n| CLIP-I | 0.9511 | 0.9232 | 0.9333 | **0.8701** | 0.9407 | 0.9319 | 0.9434 | 0.9599 |\n| CLIP-T | 0.3188 | 0.3194 | 0.3181 | **0.3108** | 0.3182 | 0.3177 | 0.3151 | 0.3175 |\n| Harmonic mean | 0.5649 | 0.5460 | 0.5537 | **0.4974** | 0.5564 | 0.5472 | 0.5546 | 0.5665 |\n| ***[Semseg eval]*** |  |  |  |  |  |  |  |  |\n| PascalVOC | 72.6 | 82.8 | 82.8 | **84.7** | 83.4 | 82.9 | 78.3 | 75.3 |\n\n**Flux.1-dev**\n\n| Layer | 0 | 2 | 6 | 8 | 9 | 10 | 12 | 15 | 17 | 26 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ***[Value norm]*** |  |  |  |  |  |  |  |  |  |  |\n| Image tokens | 57.1 | 85.8 | 41.9 | 54.8 | 52.8 | 53.6 | 56.7 | 88.9 | **89.3** | 70.6 |\n| Text tokens | 63.4 | **91.9** | 69.8 | 70.8 | 67.9 | 74.1 | 66.3 | 58.0 | 70.1 | 56.7 |\n| Prompt tokens | 72.4 | 106.3 | 104.4 | 115.1 | 103.8 | 120.6 | 127.1 | 108.6 | **139.8** | 57.3 |\n| ***[Perturbation metric]*** |  |  |  |  |  |  |  |  |  |  |\n| DINO | 0.9737 | 0.9922 | 0.9947 | 0.9829 | 0.9780 | 0.9635 | **0.9524** | 0.9820 | 0.9617 | 0.9699 |\n| CLIP-I | 0.9899 | 0.9972 | 0.9978 | 0.9938 | 0.9906 | 0.9881 | **0.9832** | 0.9946 | 0.9896 | 0.9920 |\n| CLIP-T | 0.3022 | 0.3038 | 0.3029 | 0.3031 | 0.3039 | 0.3040 | **0.3013** | 0.3036 | 0.3044 | 0.3033 |\n| Harmonic mean | 0.5611 | 0.5657 | 0.5650 | 0.5636 | 0.5636 | 0.5618 | 0.5569 | 0.5641 | **0.5546** | 0.5622 |\n| ***[Semseg eval]*** |  |  |  |  |  |  |  |  |  |  |\n| PascalVOC | 65.9 | 68.0 | 70.0 | 73.8 | 73.3 | 71.6 | 79.7 | 74.5 | **83.5** | 77.0 |\n\nFrom the table above, we can similarly observe the correlation between value norm and segmentation performance among layers for both Flux and Stable Diffusion 3.5. While SD3.5 appears to have layer 9, identical to SD3, to exhibit strong semantic grounding, Flux shows layer 12 and 17 to have a similar tendency. This hints that our observation and methodology are not proprietary for SD3, but can be applied to other DiT-based diffusion models with multi-modal attention, highlighting the generalizability of our insights and findings.\n\n### **Ablation on the choice of timestep**\n\nInitially, we chose **t=14** as our early experiments on ADE20K showed the best result for t=14 when we conducted a coarse search over the timesteps. To further explore the effects of timesteps, we evaluate over **all** 28 timesteps in VOC and COCO-Object, and present the results below. Note that we only show some of the best and worst performing layers for brevity.\n\n| **Dataset \\ Timestep** | **1** | **2** | **6** | **7** | **8** | **14** | **21** | **28** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| **Pascal VOC** | 73.9 | 84.5 | **89.2** | 89.0 | **89.2** | 87.8 | 81.0 | 70.5 |\n| **COCO-Object** | 43.9 | 55.4 | 62.0 | **62.1** | **62.1** | 58.2 | 48.5 | 25.4 |\n\nThe results reveal that t=8, in fact, consistently outperforms our original choice of t=14 for both VOC and COCO-Object, with t=6 and 7 similarly showing strong results. While we default the timestep to t=14 for additional experiments presented in the rebuttals to avoid confusion, we thank the reviewers for pointing this out and promise to update the scores as well as include the ablation on timesteps in the revised manuscript.\n\n\n### **Comparison with COCO-trained model**\n\nWe agree that the masks predicted by SAM can be overly fine‑grained, which can hinder training by enforcing the attention maps within the diffusion model to very small regions. In this regard, we carry out the same training with COCO images and masks, and report the results in weakly and unsupervised settings, as well as the generation performance with Clipscore below.\n\n|  |  | **Weakly** |  | **Unsupervised** |  | **Generation** |\n| --- | --- | --- | --- | --- | --- | --- |\n| Model | Dataset | VOC | COCO-Obj | Cityscapes | COCO-Stuff-27 | Clipscore |\n| Ours (Zero-shot) | - | 86.4 | 58.0 | 24.2 | 49.7 | 27.1437 |\n| Ours (Trained) | SA-1B | 87.0 | 58.0 | 24.2 | 50.8 | 27.2447 |\n| Ours (Trained) | COCO | **88.7** | **59.2** | **24.4** | **53.5** | **27.2760** |\n\nWe can observe that not only both weakly and unsupervised performance showed improvements, but also the generation performance has improved when compared to being trained on SA-1B. This suggests the importance of mask granularity in our fine‑tuning framework. We appreciate the reviewer’s constructive suggestion and will incorporate the COCO‑trained model into the manuscript upon revision.\n\n### **Comparison with ODISE and SemFlow**\n\n| **Method** | **Dataset** | **Supervision** | **VOC20** | **Object** | **PC59** | **ADE** | **City** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ***[Closed-set]*** |  |  |  |  |  |  |  |\n| SemFlow | COCO-Stuff | Label + mask | N/A | N/A | N/A | N/A | N/A |\n| ***[Fully-supervised]*** |  |  |  |  |  |  |  |\n| ODISE | COCO-Panoptic | Label + mask | - | - | 57.3 | 29.9 | - |\n| ***[Weakly-supervised/ Zero-shot]*** |  |  |  |  |  |  |  |\n| Ours (Zero-shot) | - | *None* | 86.4 | 58.0 | 46.1 | 31.9 | 19.0 |\n| Ours (Trained) | SA-1B | mask | 87.0 | 58.0 | 46.3 | 32.0 | 19.2 |\n| Ours (Trained) | COCO | mask | 88.7 | 59.2 | 48.2 | 32.2 | 19.7 |\n\nWe appreciate the reviewer for suggesting ODISE [1] and SemFlow [2] for comparison. While both models are based on diffusion models, we notice that both ODISE and SemFlow are trained with different levels of supervision and evaluated in different settings, which prevents a fair comparison to our work. SemFlow is trained in a closed-set manner, which cannot perform open-vocabulary segmentation, and ODISE requires full supervision of having class-labeled masks. By contrast, our model operates in a *weakly supervised, zero-shot* setting and—without any class labels, while lightweight fine-tuning with inexpensive SA-1B or COCO masks pushes the scores. Nonetheless, we show comparisons in the table above as a reference, and we will add clarifications regarding the different settings of different methods to the manuscript.\n\n[1] Xu, Jiarui, et al. \"Open-vocabulary panoptic segmentation with text-to-image diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[2] Wang, Chaoyang, et al. \"Semflow: Binding semantic segmentation and image synthesis via rectified flow.\" Advances in Neural Information Processing Systems 37 (2024): 138981-139001.\n\n\n### **Limitations of our method**\n\nWe apologize for not including the limitations of our method in the supplementary material. To address this oversight, we provide the limitations below, which will certainly be included in our revised version.\n\nOur study deliberately focuses on dense perception and image generation tasks, leaving reasoning-centric evaluations (e.g., action recognition or spatial-relations QA) to future work because our supervision signal targets segmentation semantics rather than high-level reasoning. In addition, we attach the loss to a single “sweet-spot” layer per backbone for maximal simplicity; while this already delivers strong gains, a more exhaustive layer/timestep sweep—or the addition of auxiliary heads such as optical flow or pose—could yield incremental improvements without altering our core conclusions."}, {"review_id": "u1lL76uFS3", "reviewer": "Reviewer_wi3R", "summary": "The manuscript analyse attention maps of MM-DiT, focusing on the alignment between semantic information in images and text tokens. From the analysis the authors identity the 9th layer to have higher semantic relevance, and to align text and images. Following this study, the authors inject LoRA layer in the network to improve adherence to the prompt. These layers are fine-tuned using the flow matching loss and a mask loss that encourages the alignment and segmentation text-image. \n\nThe results show that the alignment is an emergent property in these models, and the proposed approach has comparable results with baselines.", "questions": "1. Intuitively, do you expect these conclusion to be general to other MM-DiT models? (of course the layers may be different, it might be 9th and 15th)\n2. Do you expect that such a supervision would benefit a model trained from scratch? There would be no need of LoRAs and we would have to manually select layers instead of a study \n3. Given that perturbing some attention maps (Sec 3.4) does not have a major impact in the results, have the authors thought of removing I2T attentions for those layers?\n4. How would the method compare (quantitatively) with non-diffusion approaches? Did the authors run any preliminary experiments in the early stage of the project?", "limitations": "The authors fail to adequately highlight limitations and future work of the proposed work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- the quantitative results show competitive/minimal drop performance wrt the selected baselines\n- the qualitative results show better alignment and accurate generation than baselines\n- the analysis is valuable and can help designing better and more efficient DiT models\n\nWeaknesses:\n- ground truth masks are necessary, which restricts the applicability of these methods\n- the need of a diffusion model able to segment images seems limited\n- the manuscript lacks strong baselines, such as \"Segment Anything\", \"Diffuse, Attend, and Segment\", or in general non-diffusion baselines (other examples [1,2])\n\n[1] ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation\n[2] CorrCLIP: Reconstructing Correlations in CLIP with Off-the-Shelf Foundation Models for Open-Vocabulary Semantic Segmentation", "was_revised": true, "final_justification": "The authors addressed all my concerns, and I am convinced there is merit in this analysis", "rebuttal": "### **Generalizability to different models**\n\nDue to the character limit, we kindly refer the reviewer to the \"Generalizability to different models\" in the rebuttal for reviewer FeAF. We thank the reviewer for the understanding.\n\n### **Clarification on the method and the task**\nOur framework offers both a training-free path and a fine-tuning stage, neither of which **do not require ground-truth masks**—at training or inference. This lets us evaluate in zero-shot and open-vocabulary settings without sacrificing scope. When adaptation is desired, we fine-tune on pseudo-labels generated automatically by Segment Anything (SAM), so the method remains label-free and readily transferable to new domains and datasets.\n\nExtracting high-quality segments directly from diffusion models is particularly valuable: it turns the creation of otherwise costly segmentation labels into a by-product of sampling foundation models. Our study therefore focuses on uncovering the emergent segmentation behavior of MM-DiT diffusion models and on showing how a lightweight fine-tune can efficiently harness these properties for downstream tasks.\n\n### **Comparison with more baselines**\n\nWe thank the reviewer for emphasizing the need to include strong baselines in our comparison. Below, we clarify our evaluation choices:\n\n**Segment Anything (SAM)**\nWe did not include SAM [1] because its prompt-driven, open-set mask-generation paradigm is fundamentally different from our class-conditional diffusion approach and thus out of scope as a peer baseline. Instead, SAM serves as an *upper bound* on achievable mask quality.\n\n**Diffuse, Attend, and Segment**\nWe kindly invite the reviewer to **revisit Table 2**, where the “Diffuse, Attend and Segment [2]” is **already one of our core comparators**: its performance appears as a name of “DiffSeg”.\n\n**CLIP-based non-diffusion methods**\nWe would like to highlight that **CLIP-based methods are not fair to compare with ours**, where CLIP-based evaluations [3,4] assume *no* ground-truth class labels given, whereas diffusion-based methods are evaluated *with* ground-truth class names provided. This difference in setting comes from the inherent characteristics of CLIP and diffusion models, where CLIP needs distractors, while diffusion models need highly descriptive information regarding the image.\n\nAlthough this setting mismatch precludes a perfectly fair head-to-head comparison, we provide a comparison below:\n\n| **Dataset** | **VOC20** | **Object** | **PC59** | **ADE** | **City** |\n| --- | --- | --- | --- | --- | --- |\n| ***[CLIP-based methods]*** |  |  |  |  |  |\n| ProxyCLIP | 83.3 | 49.8 | 39.6 | 24.2 | 42.0 |\n| CorrCLIP | 91.8 | 52.7 | 47.9 | 28.8 | 49.9 |\n| ***[Diffusion-based methods]*** |  |  |  |  |  |\n| DiffSegmenter | 82.6 | - | - | - | - |\n| iSeg | 82.9 | 57.3 | - | - | - |\n| Ours (Zero-shot, t=14) | 86.4 | 58.0 | 46.1 | 31.9 | 19.0 |\n| Ours (Zero-shot, t=8) | 89.2 | 62.0 | 49.0 | 34.2 | 26.5 |\n\n[1] Kirillov, Alexander, et al. \"Segment anything.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[2] Tian, Junjiao, et al. \"Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Lan, Mengcheng, et al. \"Proxyclip: Proxy attention improves clip for open-vocabulary segmentation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[4] Zhang, Dengke, Fagui Liu, and Quan Tang. \"Corrclip: Reconstructing correlations in clip with off-the-shelf foundation models for open-vocabulary semantic segmentation.\" arXiv preprint arXiv:2411.10086 (2024).\n\n### **Benefit of our method for from-scratch training**\n\nWe thank the reviewer for suggesting a valuable point for discussion. Although we can not afford the computational budget to train a full diffusion model from scratch, we believe that our training strategy could benefit the model even when training from scratch. Given that we are able to improve both generation and segmentation performance, we consider our method to provide training signals that align with the diffusion model’s knowledge. A recent work, REPA [5] trains diffusion networks from scratch without LoRA, while enforcing a feature‑alignment loss against DINO. This direct supervision accelerates convergence **and** improves generation fidelity, showing that alignment losses can be effective even when the entire model is updated.\nNonetheless, REPA also shows that the layer selection for applying the alignment loss is crucial for both sample quality and training speed. Similarly, we expect layer-selection to still be of importance as it seems crucial to maintain alignment with the layer that naturally shows emergent properties, in terms of training efficiency, generation quality, and segmentation capabilities. We will add such discussions to the paper upon revision.\n\n[5] Yu, Sihyun, et al. \"Representation alignment for generation: Training diffusion transformers is easier than you think.\" arXiv preprint arXiv:2410.06940 (2024).\n\n### **Additional results on the perturbation experiment**\n\nTo probe the importance of each cross‑attention block, we repeated the perturbation study but *fully masked* the image‑to‑text (I2T) channels one layer at a time. Qualitatively, removing I2T attention produces artifacts that mirror those seen when we previously applied a Gaussian blur to the same pathway, indicating comparable disruption mechanisms. To assess the effects in a quantitative manner, we adopted DreamBooth’s protocol [7] for generating 50 images per perturbation and reported the average change in DINO, CLIP‑I, and CLIP‑T scores relative to the unperturbed baseline. All three metrics dip most sharply when **layer 9** is masked, underscoring its pivotal role in aligning text prompts with visual content.\n\n| **Layer** | **L3** | **L7** | **L8** | **L9** | **L12** | **L13** | **L15** | **L16** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DINO Score | 0.6933 | 0.6682 | 0.6816 | **0.4919** | 0.6665 | 0.6370 | 0.6779 | 0.6691 |\n| CLIP-I Score | 0.8898 | 0.8783 | 0.8772 | **0.8305** | 0.8737 | 0.8585 | 0.8830 | 0.8843 |\n| CLIP-T Score | 0.2912 | 0.2863 | 0.2844 | **0.2786** | 0.2930 | 0.2855 | 0.2908 | 0.2886 |\n| Harmonic mean | 0.4999 | 0.4895 | 0.4899 | **0.4394** | 0.4952 | 0.4809 | 0.4961 | 0.4925 |\n\nWe thank the reviewer for the valuable suggestion, and we will include the qualitative and quantitative results as well as the discussions in the final version.\n\n[7] Ruiz, Nataniel, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n### **Limitations of our method**\n\nWe apologize for not including the limitations of our method in the supplementary material. To address this oversight, we provide the limitations below, which will certainly be included in our revised version.\n\nOur study deliberately focuses on dense perception and image‐generation tasks, leaving reasoning-centric evaluations (e.g., action recognition or spatial-relations QA) to future work because our supervision signal targets segmentation semantics rather than high-level reasoning. In addition, we attach the loss to a single “sweet-spot” layer per backbone for maximal simplicity; while this already delivers strong gains, a more exhaustive layer/timestep sweep—or the addition of auxiliary heads such as optical flow or pose—could yield incremental improvements without altering our core conclusions."}, {"review_id": "cekZeOEiFR", "reviewer": "Reviewer_3ee8", "summary": "This paper explores the attention mechanism of MM-DiT in diffusion models, focusing on how specific heads and layers propagate semantic information and influence generation quality. This paper identifies critical layers in MM-DiT that reveal strong influences on image segmentation and generation. A corresponding fine-tuning-based and training-free method is proposed for enhancing the segmentation and generation qualities.", "questions": "See Weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The insight into how attention maps influence segmentation and generation is interesting.\n\n2. The writing style is easy to follow.\n\nWeaknesses:\n\n1. This paper discusses the attention map of the MMdit architecture but is limited to SD3. The authors are encouraged to explore MM-DiT-based models, such as FLUX [1], to demonstrate the generalizability of the findings.\n\n2. There have been some works discussing the application of mutual attention maps and self-attention maps in image segmentation. I am confused about the difference between this paper and previous works[2,3]. In addition, in Tab.2, the performance of Diff4Seg is significantly lower than [2].\n\n3. Some expressions are not unified. For example, L263 uses the abbreviation `Tab.2`, but L250 uses the full name `Table 1`. \n\n[1] https://github.com/black-forest-labs/flux\n\n[2] Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. CVPR, 2024.\n\n[3] Diffusion model is secretly a training-free open vocabulary semantic segmenter. TIP, 2025.", "was_revised": true, "final_justification": "I think the main issues have been addressed, and the update score is 4. Since the method's sensitivity to targets of varying sizes is unclear, the authors should discuss these limitations in the final version and provide the necessary numerical results.", "rebuttal": "### **Generalizability to different models**\n\nTo demonstrate that our insights are not tied to SD3, we replicated the entire analysis on two additional MM‑DiT variants—SD 3.5 [1] and Flux [2]. Summarizing the results, we observed consistent trends across all models (Note that the exact layer‑wise statistics can differ between models).\n\nBecause the rebuttal cannot include external media, we present the key quantitative results here and will supply the corresponding plots and qualitative examples in the camera‑ready version. Following the methodology in the main paper, we use three complementary evaluation protocols:\n\n- **Attention‑norm analysis** – We compute the value‑vector norms for image, text, and actual prompt tokens at every layer, highlighting only the most informative layers for readability.\n- **Attention‑perturbation analysis** – For each layer, we perturb its cross‑attention and measure DINO, CLIP‑I, and CLIP‑T similarities between the original and perturbed images; larger drops indicate layers that are critical to image fidelity.\n- **Segmentation evaluation** – Using cross‑attention maps as pseudo‑masks, we evaluate mIoU on Pascal VOC to assess mask quality.\n\nIn the table, bold indicates the highest entry for value norm and segmentation metrics, and the lowest for perturbation metrics.\n\n**Stable Diffusion 3.5 Medium**\n\n| Layer | 3 | 7 | 8 | 9 | 12 | 13 | 15 | 16 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ***[Value norm ratio]*** |  |  |  |  |  |  |  |  |\n| Image tokens | **48.5** | 39.5 | 39.1 | 27.7 | 35.8 | 32.1 | 37.3 | 42.8 |\n| Text tokens | 17.4 | 29.3 | 29.8 | **41.5** | 27.4 | 41.2 | 33.3 | 23.2 |\n| Prompt tokens | 22.9 | 35.6 | 37.3 | **53.8** | 37.0 | 47.2 | 44.8 | 32.0 |\n| ***[Perturbation metric]*** |  |  |  |  |  |  |  |  |\n| DINO | 0.8910 | 0.7814 | 0.8314 | **0.6011** | 0.8432 | 0.7929 | 0.8507 | 0.9056 |\n| CLIP-I | 0.9511 | 0.9232 | 0.9333 | **0.8701** | 0.9407 | 0.9319 | 0.9434 | 0.9599 |\n| CLIP-T | 0.3188 | 0.3194 | 0.3181 | **0.3108** | 0.3182 | 0.3177 | 0.3151 | 0.3175 |\n| Harmonic mean | 0.5649 | 0.5460 | 0.5537 | **0.4974** | 0.5564 | 0.5472 | 0.5546 | 0.5665 |\n| ***[Semseg eval]*** |  |  |  |  |  |  |  |  |\n| PascalVOC | 72.6 | 82.8 | 82.8 | **84.7** | 83.4 | 82.9 | 78.3 | 75.3 |\n\n**Flux.1-dev**\n\n| Layer | 0 | 2 | 6 | 8 | 9 | 10 | 12 | 15 | 17 | 26 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ***[Value norm]*** |  |  |  |  |  |  |  |  |  |  |\n| Image tokens | 57.1 | 85.8 | 41.9 | 54.8 | 52.8 | 53.6 | 56.7 | 88.9 | **89.3** | 70.6 |\n| Text tokens | 63.4 | **91.9** | 69.8 | 70.8 | 67.9 | 74.1 | 66.3 | 58.0 | 70.1 | 56.7 |\n| Prompt tokens | 72.4 | 106.3 | 104.4 | 115.1 | 103.8 | 120.6 | 127.1 | 108.6 | **139.8** | 57.3 |\n| ***[Perturbation metric]*** |  |  |  |  |  |  |  |  |  |  |\n| DINO | 0.9737 | 0.9922 | 0.9947 | 0.9829 | 0.9780 | 0.9635 | **0.9524** | 0.9820 | 0.9617 | 0.9699 |\n| CLIP-I | 0.9899 | 0.9972 | 0.9978 | 0.9938 | 0.9906 | 0.9881 | **0.9832** | 0.9946 | 0.9896 | 0.9920 |\n| CLIP-T | 0.3022 | 0.3038 | 0.3029 | 0.3031 | 0.3039 | 0.3040 | **0.3013** | 0.3036 | 0.3044 | 0.3033 |\n| Harmonic mean | 0.5611 | 0.5657 | 0.5650 | 0.5636 | 0.5636 | 0.5618 | 0.5569 | 0.5641 | **0.5546** | 0.5622 |\n| ***[Semseg eval]*** |  |  |  |  |  |  |  |  |  |  |\n| PascalVOC | 65.9 | 68.0 | 70.0 | 73.8 | 73.3 | 71.6 | 79.7 | 74.5 | **83.5** | 77.0 |\n\nFrom the table above, we can similarly observe the correlation between value norm and segmentation performance among layers for both Flux and Stable Diffusion 3.5. While SD3.5 appears to have layer 9, identical to SD3, to exhibit strong semantic grounding, Flux shows layer 12 and 17 to have a similar tendency. This hints that our observation and methodology are not proprietary for SD3, but can be applied to other DiT-based diffusion models with multi-modal attention, highlighting generalizability of our insights and findings.\n\n\n### **Comparison with previous attention-based methods**\n\nWhile we acknowledge that performing segmentation based on attention maps of diffusion models is an existing idea, we highlight that our exploration mainly focuses on understanding MM-DiT-based diffusion models and explore emergent properties and reveal that certain layers are able to show strong segmentation capabilities. On the other hand, [1,2] focuses on gathering information within the attention maps to identify segments, without fully regarding the image-text interaction within the diffusion models. This requires iterative algorithms [1] or tuning highly sensitive hyperparameters [2], whereas our method can simply obtain segmentation directly from a single attention layer and show competitive results. We also highlight that despite being simple, we outperform [2] in terms of IoU, while having a deficit in accuracy.\n\nWe wish to highlight that our key contribution lies more on the in-depth analysis and investigation of learned representations of MM-DiT, exploration of critical lays for segmentation capabilities, and demonstration of strong zero-shot segmentation performance with further possible enhancement using localization capabilities of the critical layers.\n\n[1] Tian, Junjiao, et al. \"Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Wang, Jinglong, et al. \"Diffusion model is secretly a training-free open vocabulary semantic segmenter.\" IEEE Transactions on Image Processing (2025).\n\n### **Unified expressions**\n\nThank you for pointing out the inconsistency in expressions. We will update our manuscript to unify those accordingly.\n\n### **Limitations of our method**\n\nWe apologize for not including the limitations of our method in the supplementary material. To address this oversight, we provide the limitations below, which will certainly be included in our revised version.\n\nOur study deliberately focuses on dense perception and image‐generation tasks, leaving reasoning-centric evaluations (e.g., action recognition or spatial-relations QA) to future work because our supervision signal targets segmentation semantics rather than high-level reasoning. In addition, we attach the loss to a single “sweet-spot” layer per backbone for maximal simplicity; while this already delivers strong gains, a more exhaustive layer/timestep sweep—or the addition of auxiliary heads such as optical flow or pose—could yield incremental improvements without altering our core conclusions."}, {"review_id": "CUoF7VPtNv", "reviewer": "Reviewer_tfvR", "summary": "This paper investigates how semantic information is structured and propagated within Multi-Modal Diffusion Transformers (MM-DiTs) by analyzing their attention mechanisms. The authors identify specific attention heads and layers that align text tokens with spatially coherent image regions, contributing to both image generation and zero-shot segmentation. They propose a lightweight LoRA-based fine-tuning approach to enhance these capabilities without compromising image quality. The study reveals that semantic alignment is an emergent property of diffusion transformers and can be selectively strengthened to improve both generation and perception tasks.", "questions": "- Could you provide additional qualitative comparisons between the baseline and fine-tuned models in the generation setting to better illustrate semantic or visual differences?\n- ZestGuide [3] proposes a mechanism that seems similar to the one introduced in Figure 6, albeit in a zero-shot conditional generation context. Could you clarify how your approach differs? Have you considered comparing against such methods? This could further support the findings in Table 3 and reinforce the claims about improved generation.\n- Line 157 mentions reporting results at timesteps T=14 and T=28, but Appendix A.1 only refers to T=14 of 28. It is unclear where T=28 is presented. Have you conducted any ablation over timesteps to evaluate how semantic richness evolves? This could add additional insight to your analysis.\n\nIf the authors provide additional experiments, on other backbones and more benchmarks, I’ll be happy to raise my score. \n\nTypos:\n- L.171: now To &rarr; now to\n- L.187 image-text alignment,, &rarr; image-text alignment,\n- L.190 it can be observed that the only the layers &rarr; it can be observed that only the layers\n\n[3] Couairon, G., Careil, M., Cord, M., Lathuiliere, S., & Verbeek, J. (2023). Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2174-2183).", "limitations": "I don't see a discussion about the limitations of the method. Could you discuss the limitations in the supplementary material as mentioned in the checklist.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is clearly written and well-structured, making it easy to follow.\n- It addresses an underexplored area—how semantic information is stored and propagated within MM-DiT models. This is a highly relevant topic, as understanding these mechanisms can aid in designing more effective diffusion transformers with improved image-text alignment, potentially boosting generative capabilities and downstream performance.\n- Figures are well-designed and effectively convey the main findings.\n- The experimental analysis is thoughtful and yields valuable insights into the internal behavior of these models.\n\n**Weaknesses**\n- The experiments are limited to Stable Diffusion 3, which restricts the generalizability of the findings. Expanding the evaluation to models like SD3.5 or FLUX, and including a range of model scales (as in Figures 1, 2, and 4), would help assess the robustness and scalability of the approach.\n- The claim in Line 200 that the proposed method “ultimately boosts segmentation performance” is not fully supported by the results. Table 1 compares only two baselines, and Table 2 presents mixed outcomes. Although Diff4Seg shows improved mIoU, it performs worse in accuracy compared to DiffSeg on Cityscapes. The fine-tuned Diff4Seg shows only marginal gains—primarily on COCO-Stuff-27 mIoU. Additionally, DiffCut is mentioned in related work but not included in the comparison. Broader benchmarking (e.g., on VOC20, ADE20K) would strengthen this claim.\n- Similarly, the claim of improved “generation performance” (Line 200) lacks convincing evidence. In Table 3, only the FID score improves; IS and CLIP Score do not outperform the baseline. While evaluating generation quality is inherently difficult, incorporating more metrics (e.g., FD-DINOv2 [1], Attribute Binding and Spatial Relationships [2] from T2I-CompBench) and more qualitative examples could better support the claim.\n\n[1] Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., ... & Loaiza-Ganem, G. (2023). Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 3732-3784.\n\n[2] Huang, K., Sun, K., Xie, E., Li, Z., & Liu, X. (2023). T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36, 78723-78747.", "was_revised": true, "final_justification": "My initial concerns have been addressed in the rebuttal, this papers shows valuable insights into how semantic representations emerge and aggregate in DiT models, providing meaningful guidance toward unified models that bridge visual generation and perception. I increase my score to borderline accept.", "rebuttal": "### **Generalizability to different models**\nDue to the character limit, we kindly refer the reviewer to the \"Generalizability to different models\" in the rebuttal for revewer FeAF. We thank the reviewer for the understanding.\n\n### **Clarification on expressions**\n\nWe greatly appreciate the reviewer’s constructive suggestion. Below, we have conducted broader benchmarking on unsupervised segmentation settings and compared it with the suggested baseline. In addition to the SA-1B–trained model presented in the manuscript, we also include COCO-trained variants for comparison. The results are summarized below:\n\n| Model | Training dataset | VOC20 | PC59 | COCO-Obj | COCO-Stuff-27 | Cityscapes | ADE20K |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| DiffCut [1] | - | 62.0 | 54.1 | 32.0 | 46.1 | 28.4 | 42.4 |\n| Ours (Zero-shot) | - | 54.9 | 52.6 | 38.5 | 49.7 | 24.2 | 44.9 |\n| Ours (Trained) | SA1B | 55.1 (+0.2) | 52.8 (+0.2) | 39.0 (+0.5) | 50.8 (+1.1) | 24.2 (0.0) | 45.0(+0.1) |\n| Ours (Trained) | COCO | 56.1 (+1.2) | 53.5 (+0.9) | 38.8 (+0.3) | 53.5 (+3.8) | 24.4 (+0.2) | 45.4 (+0.5) |\n\nThese results confirm that the insights from our layer-selection analysis translate into tangible improvements across multiple datasets, and they also invite future work on more sophisticated supervisory designs.\n\nIn light of the reviewer’s comment, we will replace “*ultimately boosts both segmentation and generation performance*” with a more precise statement: *“The proposed supervision improves segmentation accuracy without degrading, and in some cases modestly enhances, image generation quality.”* We apologise for any earlier overstatement.\n\n[1] Couairon, Paul, et al. \"Diffcut: Catalyzing zero-shot semantic segmentation with diffusion features and recursive normalized cut.\" Advances in Neural Information Processing Systems 37 (2024): 13548-13578.\n\n### **Regarding improvement in generation quality**\n\nWe evaluate the CLIP score to assess the alignment between generated images and their corresponding text prompts. To ensure robustness against randomness, we conduct evaluations on three diverse prompt sets: (1) 500 prompts from Pick-a-Pic [2], with five images generated per prompt; (2) 5,000 images generated using prompts from the COCO caption validation set; and (3) captions generated by CogVLM for 1,000 validation images sampled from SA-1B, which do not overlap with our training data.\n\n| **clipscore ↑** | **Training dataset** | **Pick-a-Pic** | **COCO** | **SA-1B** | **Mean** |\n| --- | --- | --- | --- | --- | --- |\n| **Baseline** | - | 27.0252 | 26.0638 | 28.3422 | 27.1437 |\n| **Ours (Trained)** | SA1B | **27.0547** | 26.2318 | 28.4476 | 27.2447 |\n| **Ours (Trained)** | COCO | 27.0409 | **26.2319** | **28.5553** | **27.2760** |\n\nBased on the reviewer’s feedback, we also evaluated our model on T2I-Compbench++ [3].\n\n| **T2I compbench++** | **Training dataset** | **color** | **shape** | **texture** | **2D spatial** | **3D spatial** | **numeracy** | **non spatial** | **complex** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| **Baseline** | **-** | 0.7864 | 0.5644 | 0.7200 | **0.2435** | **0.3318** | 0.5566 | **0.3124** | 0.3719 |\n| **Ours (Trained)** | **SA1B** | 0.7836 | 0.5679 | 0.7252 | 0.2330 | 0.3151 | 0.5460 | 0.3113 | 0.3709 |\n| **Ours (Trained)** | **COCO** | **0.7919** | **0.5687** | **0.7260** | 0.2301 | 0.3234 | **0.5584** | 0.3120 | **0.3735** |\n\nOverall, our method outperforms the baseline on nearly every metric. The only exceptions are the reasoning-focused benchmarks—specifically the 2-D/3-D spatial and non-spatial tests—where our scores trail the baseline. This shortfall is unsurprising: those tasks demand explicit spatial reasoning and action understanding, whereas our supervisory signal was designed to strengthen perceptual and segmentation cues, not high-level reasoning. In light of the reviewer’s comment, we will also rephrase the claims by slightly toning down, if the reviewer agrees. We apologise for any earlier overstatement.\n\nAlso, we will conduct and include a user study to better assess the generation quality of ours in the final version.\n\n[2] Kirstain, Yuval, et al. \"Pick-a-pic: An open dataset of user preferences for text-to-image generation.\" Advances in neural information processing systems 36 (2023): 36652-36663.\n\n[3] Huang, Kaiyi, et al. \"T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.\" Advances in Neural Information Processing Systems 36 (2023): 78723-78747.\n\n### **Comparison with ZestGuide**\nWe thank the reviewer for suggesting discussions regarding ZestGuide [4]. While both ZestGuide and ours leverage similar mask loss design, we notice that as ZestGuide targets layout generation, its goal is to enforce the diffusion model to generate the instance in the mask area. In contrast, our motivation is to improve multi-modal interaction within certain layers, rather than enforcing certain objects to be grounded to certain locations. This also differentiates our work with ZestGuide in terms of training configuration, where we generally fine-tune the model in a set of images, whereas ZestGuide requires per-image optimization. Nonetheless, we appreciate the reviewer’s pointer and remain open to further suggestions.\n\n[4] Couairon, Guillaume, et al. \"Zero-shot spatial layout conditioning for text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n### **Regarding the choice of timestep**\n\nWe apologize for the typo in L157: the text should read “**T = 14 out of 28**”, not “T = 14 and T = 28”. We will correct this in the revision.\n\nInitially, we chose **t=14** as our early experiments on ADE20K showed the best result for t=14 when we conducted a coarse search over the timesteps. To further explore the effects of timesteps, we evaluate over **all** 28 timesteps in VOC and COCO-Object, and present the results below. Note that we only show some of the best and worst performing layers for brevity.\n\n| **Dataset \\ Timestep** | **1** | **2** | **6** | **7** | **8** | **14** | **21** | **28** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| **Pascal VOC** | 73.9 | 84.5 | **89.2** | 89.0 | **89.2** | 87.8 | 81.0 | 70.5 |\n| **COCO-Object** | 43.9 | 55.4 | 62.0 | **62.1** | **62.1** | 58.2 | 48.5 | 25.4 |\n\nThe results reveal that t=8 in fact consistently outperforms our original choice of t=14 for both VOC and COCO-Object, with t=6 and 7 similarly showing strong results. While we default the timestep to t=14 for additional experiments presented in the rebuttals to avoid confusion, we thank the reviewers for pointing this out and promise to update the scores as well as include the ablation on timesteps in the revised manuscript.\n\n### **Typos**\n\nThank you for pointing out the typos. We apologize for any confusion caused by this and will update our manuscript accordingly.\n\n### **Limitations of our method**\n\nWe apologize for not including the limitations of our method in the supplementary material. To address this oversight, we provide the limitations below, which will certainly be included in our revised version.\n\nOur study deliberately focuses on dense perception and image‐generation tasks, leaving reasoning-centric evaluations (e.g., action recognition or spatial-relations QA) to future work because our supervision signal targets segmentation semantics rather than high-level reasoning. In addition, we attach the loss to a single “sweet-spot” layer per backbone for maximal simplicity; while this already delivers strong gains, a more exhaustive layer/timestep sweep—or the addition of auxiliary heads such as optical flow or pose—could yield incremental improvements without altering our core conclusions."}]

Author rebuttal:

Meta-review: This paper analyses the structure of attention maps of MM-DiTs, to study the alignment between semantic information in image and text tokens. Following this analysis, the paper proposes a lightweight LoRA-based fine-tuning to improve controlability by addressing the relevant attention heads. The proposed approach is based on an interesting analysis and yields results comparable to respective baselines, while improving alignment.

The reviews agree that the analysis is valuable and can help designing better and more efficient MM-DiTs. Initial criticism on the method limitations, and limited evaluation, for example only on SD3, has been discussed and the authors have provided significant additional analysis and results in the rebuttal. Therefore, after the discussion, all reviewers agree that major concerns have been addressed and recommend the paper for acceptance. The AC agrees.
Decision: Accept (poster)