Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations
Authors: Chaofan Gan
Yuanpeng Tu
Xi Chen
Tieyuan Chen
Yuxi Li
Mehrtash Harandi
Weiyao Lin
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Transformers
Massive Activations
Representation Learning
Primary area: applications

Abstract: Pre-trained stable diffusion models (SD) have shown great advances in visual correspondence. 
In this paper, we investigate the capabilities of Diffusion Transformers (DiTs) for accurate dense correspondence. Distinct from SD, DiTs exhibit a critical phenomenon in which very few feature activations exhibit significantly larger values than others, known as massive activations, leading to uninformative representations and significant performance degradation for DiTs.
The massive activations consistently concentrate at very few fixed dimensions across all image patch tokens, holding little local information. 
We analyze these dimension-concentrated massive activations and uncover that their concentration is inherently linked to the Adaptive Layer Normalization (AdaLN) in DiTs.
Building on these findings, we propose the Diffusion Transformer Feature (DiTF), a training-free AdaLN-based framework that extracts semantically discriminative features from DiTs. 
Specifically, DiTF leverages AdaLN to adaptively localize and normalize massive activations through channel-wise modulation. 
Furthermore, a channel discard strategy is introduced to mitigate the adverse effects of massive activations.
Experimental results demonstrate that our DiTF outperforms both DINO and SD-based models and establishes a new state-of-the-art performance for DiTs in different visual correspondence tasks (e.g., with +9.4\% on Spair-71k and +4.4\% on AP-10K-C.S.).

Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations
Chaofan Gan 1.2
Yuanpeng Tu Xi Chen
Tieyuan Chen
Yuxi Li
Mchrtash Harandì Weiyao Lin
Shanghai Jiao Tong University Monash University
2The University of Hong Kong
{ganchaofan, tieyuanchen, lyxok1, wylin] edu.. cn {chaofan gan, mehrtash harandi)@monash, edu {tjtuyuanpeng, xichen. csaicsaaaaa..... maaii/mgmi..com
Abstract
Pre-trained stable diffusion models (SD) have shown great advances visual corre- spondence. In this paper, we investigate the capabilities of Diffusion Transformers (DiTs) for accurate dense correspondence. Distinct from SD, DiTs exhibit criti- cal phenomenon in which very few feature activations exhibit significantly larger values than others, known as massive activations, leading to uninformative represen- tations and significant performance degradation for DiT'. The massive activations consistently concentrate at very few fixed dimensions across all image patch tokens, holding little local information. We analyze these ciamensiooncoocenttrann activations and uncover that their concentration inherently linked to the Adaptive Layer Normalization (AdaLN) in DiT's. Building these findings, propose the Diffusion Transformer Feature (DiTF), a training-free AdaL.--based framework that extracts semantically discriminative features from DiTs. Specifically, DiTF leverages AdaLN to adaptively localize and normalize massive activations through channel-wise modulation. Furthermore, a channel discard strategy is introduced mitigate the adverse effects of massive activations. Experimental results demon- strate that our DiTF outperforms both DINO and SD-based models and establishes a new state-of soaiie--f-t---att -feeharrtt performance for DiTs in different visual correspondence tasks (c.g.. with +9.4% on Spair-7lk 7k and +4.4% on AP-IOK-C.S))
massive
90
Original feature
Modulating feature with AdaLN
Original
80
Meslolainn, Sustare with AdvEN
70
60 g01-
50
40
30
Dia
Dise
20
nokess
talens
10
Alsam--tahk
(4) Activation magnitudes
Correspondence
Figure 1: AdaLN enhances DiT features by mitigating massive activations. Original DiT features show concentrated massive activations. (b) Semantic correspondence performance using different features. Original DiT features yield poor performance due to the presence of massive acti- vations. By modulating these activations, AdaLN significantly boosts correspondence performance.
Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input Image
Pixart-Alpha
SD3-5
Figure 2: Massive activations in Diffusion Transformers (iiss). We visualize the activation magnitudes (z-axis) of features across various diffusion models. Unlike the Stable Diffusion (SD2-1) model, all DiTs exhibit distinctive phenomenon where very few feature activations show significantly higher activation values, more than 100 times larger than others. We refer to this phenomenon as massive activations, which has also been observed large language models (LLMs) 20
Introduction
Detecting dense correspondence between images plays important role in various vision tasks such
as segmentation 2 4. image editing S 6 77. and point tracking 8 10 [1]. Various works [12]
B 14 utilize pre-trained self-supervised models like DINOv2 [12 to extract image features, which
are then adapted for precise semantic correspondence. With the advent of text-to-image (T2I) generative models [5 [6], recent studies [17] [4] have demonstrated the powerful capability of pre-trained SD 15 as an effective feature extractor [14
18
19
for visual correspondence.
Compared with SD, recent DiT' (e.g., Pixart-alpha [1]], SD3 [22] and Flux (23]) demonstrate great superiority in scalability, robustness, model efficiency, and generation quality. While pre-trained SD models have been successfully leveraged feature extractors, the potential of DiTs for visual perception tasks remains largely unexplored. To bridge this gap, we take the first step to investigate the feasibility of using pre-trained DiT's feature extractors for visual correspondence tasks.
However, when directly extracting the original features for representation learning, as done SD, DiTs exhibit disappointing performance for visual perception tasks, as shown in Figure [ To better understand this discrepancy, we analyze the statistical properties of features extracted from both SD and DiTs (see Figure(). Notably, we observe that DiT feature maps exhibit massive activations, in contrast to the smoother distribution observed SD2. [55] These massive activations share few fixed dominant dimensions, resulting in a high degree directional similarity for feature vectors in the latent space (24). This property makes difficult distinguish spatial feature vectors using cosine similarity, leading to semantically uninformative and indiscriminate representations.
In this work, we aim to deepen the understanding of massive activations for representation learning, thereby enabling the extraction of semantically discriminative representations from DiTs. We begin by investigating the spatial and dimensional properties of these activations and find that they consistently emerge in small set of fixed feature dimensions across all image tokens, which carry limited local information that crucial for our task. Further analysis reveals an intriguing connection between massive activations and the AdaL in DiTs, where the concentrated dimensions of massive activations align with the residual scaling factors produced by AdaLN. To further understand this phenomenon, investigate the role of AdaLN in modulating DiT features. Our findings reveal that the built-in AdaLN effectively localizes massive activations and performs semantically meaningful thannel-wise modulation to suppress them, thereby enhancing the semantic richness and discriminative capacity of DiT features.
Based on these findings, we propose training-free AdaLN--asee framework: Diffusion Transformer Feature (DiTF), to extract structural and semantic features from DiT's Specifically, DiTF leverages built-in AdaLN in DiT's to adaptively normalize massive activations through channel-wise modulation. In addition, channel discard strategy introduced to further mitigate adverse effects of massive activations representation learning. Our main contributions are summarized as follows.

---

We identify and characterize massive activations in DiTs and find them consistently concen- trated in a small number of fixed feature dimensions across all spatial tokens.
We analyze the source of these dimensionally concentrated massive activations and uncover their strong connection to the AdaLN layer DiTs.
We propose training- feining-freee AddaLN-based framework, Diffusion Transformer Feature (DiTF), for extracting semantically discriminative representations from DiTs.
Our framework outperforms both DINO and SD-based models, establishing new state-of- the-art for Diffusion Transformers on visual correspondence tasks.
Related Work
Visual Correspondence.
Visual correspondence aims establish dense correspondence between two different images, which plays a crucial role in various visual tasks such as segmentation T
2
3
image editing [5 7 and point tracking 8 10 Early methods like HOG 223, and SIFT 26
design hand-crafted features with scale-invariant key-points and then establish image correspondences. Various works attempt to learn image correspondence in supervised learning regime. However, the supervised- based methods (27 (8 29 BI] depend on ground trrth correspondence annotations, which are challenging apply to datasets lacking precise correspondence annotations. To handle this challenge, some studies have employed DINO [12 32) and Generative Adversarial Networks (GAN) B3 extract visual features for correspondence. More recently,
34
14
35 have
demonstrated that SD models can serve as effective feature extractors for various perception tasks.
capable of producing high-quality images with reasonable structures and compositional semantics, indicating their robust spatial awareness and semantic understanding capabilities. Consequently, numerous studies have utilized pre-trained SD models as feature extractors, extracting features from images for Inspired by these works, [17] and [18] have employed the pre-trained SD model 15). as a representation learner for visual correspondence. In addition, [14] investigated the distinct properties of SD features compared to the DINO (35) feature, and jointly leveraged them for improved performance. Based on this, Gwo-wSSS
Diffusion Model. Large-scale pre-trained diffusion models 15
36 37
38
16
visual perception tasks (39 40
42
43 proposes adaptive pose
alignment improve the geometric awareness of the SD features.
Massive Activations. Massive activations have been extensively studied in large language models (LLMs) [20
44 45). Several works 20 46) have shown that these activations are concentrated in a small number of fixed dimensions, particularly in the start and delimiter tokens. [7 further revealed that the massive values in the query and key vectors originate from the positional encoding mechanism RoP)[48). In addition to LLMs, similar phenomena have also been observed in Vision Transformers (ViTs) [3 49 20). Specifically, (20) identified attention artifacts in VIT feature maps, predominantly appearing in low-information background regions. DVT 49 further traced these artifacts the influence of positional embeddings. Recent studies on accelerating DiT's (50 5T have identified massive activation outliers in DiTs and revealed that these outliers lead to unstable model quantization and excessively high distillation losses during knowledge distillation.
3 Preliminaries
Adaptive Layer Norm. AdaLN operation AdaLN(<:; y, 3) adaptively normalizes feature with the adaptive scale yy and shift 9 parameters.
2 = (1 (1+1) LayerNorm (2) +
(I)
where LayerNorm the layernorm operation [52]
DiT Architecture. We follow the architecture used in 36 DiT 36 comprises two parts: VAE (33. which comprises an encoder & and a decoder D project images from pixel space to latent space, and diffusion transformer blocks A = {AA J where k is the index and N is the number of blocks. Given an input image xo. we first encode to the latent representation 2g (aa) with the encoder E and then add corresponding noise to the latent to obtain noisy latent representation
2, E CCC according to dre-defined timestep The DiT block A& encodes the original feature
#t as follows:
21
k-1
* 2t + Aap
(2)
3

---

Faled
Pans
1000
Figure 3: Massive Activations in SD3-5. We visualize the activation magnitudes of four different image features extracted using SD3-5. Notably, massive activations consistently distribute in fixed dimension (676) across all image patch tokens.
where each Diffusion Transformer block employs residual connections [54], with ar representing residual scaling factor computed by the AdaLN-zen layer. It is important to note that the original feature throughout this paper is denoted as z/. and our analysis of massive activations is conducted based on this representation.
DiT block Ag. DiT block (36) includes modulation layer called AdaIN-zero layers (3)] which regress two groups of scale and shift parameters based on the given timestep and additional conditions (e.g., text embedding c).
{( (%
(L)
where MLP is utilized to regress channel-wise scale and shift parameters v Si of E TRC 24. Pk served as parameters for is AdaLN and a is applied to scale ieh residual connections.
Next, the block-input feature of is passed through the Self Attention layer to yield intermediate feature (k.2)
(k.1) AdaLN(=2 (k.1), (kk.)) 7 13,)))))
(4)
(k.2) Aalf-Attention (2(k:1)) 2 (k.1)
(k.1)
ot
(k.2) fed into the Feedforward layer to yield block-output feature
Finally, the intermediate feature 2t (k+1.11
(k.2) (da (k.2). :22.190)
2t
(6)
1.11 Feodforward (2) (2)222) + ax=t 22222 (A.2)
21
(7)
where 2t (R.1) and 2; to 21 (1,1) and =t (k.2) the pre-AdaLNN intermediate features, and & (k..1) and (l:22) the Post-AdaL.N
(k+1.1) denote the same features as 2t and 2, respectively. For clarity, we refer
intermediate features in the subsequent sections. The residual scaling factor as Equation 2
corresponds to az. For simplicity, we use AdaLN to refer to the AdaLN- module in DiTs
throughout this paper. See the Appendix for illustration of DiTs architecture (36).
DiTF: Diffusion Transformer Feature
In this section, explore how to eliminate the undesirable effects of massive activations and extract semantically meaningful features from DiTs. Specifically, we first analyze the spatial and dimensional characteristics of massive activations in DiTs, then discuss why the AdaLN layer is well-suited to addressing massive activation in the context of representation learning, and finally present approach for extracting semantic-diseriminatie features from DiT's
4.1 Massive Activations in Diffusion Transformers
Massive activations are high-magnitude scalar values. We first introduce quantitative criterion
characterize massive activations in DiT features. Following the definition proposed for LLMSs (20,
we adopt generalized formulation: an activation is considered massive activation if its magnitude
approximately 100 times greater than the median magnitude within feature map. is notable that massive activations scalar values, which are determined jointly by the patch and dimensions (20).

---

Table 1: Massive activations exhibit relatively low variance compared to non-massive activa- tions. We present the mean and variance of activation values across image patches for selected feature dimensions ranked by average magnitude. The Ist-ranked dimension (massive activations) shows substantially lower variance relative to its mean compared to the 10th and 20th (no-massive activations), indicating limited local information.
Model
1st
2th
10th
20th
Median
SD3-5 4..5 Flux 40.66+33.99
1.03+11.39 -3.562+8.50
-0.18=2.36 -0.41+3.16
-0.25:22.07 0.49++2.92
0.21
0.13
*++1 Original fenture 2t
Residual scale Oş
Original feature 2
Residual scale Oj
SD3-5
Flux
Figure 4: Massive activations dimensions align with the residual scaling factor ag.. We visualize the magnitudes for the original feature -++1 and residual scaling factor or.
Massive activations are present in very few fixed dimensions. We identify the dimensional locations of massive activations within the original DiT feature. As shown in Figure 3 the SD3-5 model exhibits massive activations concentrated in single feature dimension (676). Similarly, in the Flux model, massive activations appear in two specific dimensions (154 and 1446). Analyses of other DiTs are provided in the Appendix. These observations suggest that massive activations consistently occur in small number of fixed feature dimensions, which aligns with similar findings in LLMs 20
Massive activations appear across all spatial tokens. As shown in Figures and massive activations consistently appear across all image patch tokens, regardless of the input image. For instance, in the SD3-5 model, the 676th feature dimension exhibits massive activations across all patch tokens. This phenomenon differs from the observations in LLMs and Vision Transformers (ViTs). Specifically, [20] reported that massive activations in LLMs are primarily concentrated in special tokens such as start and delimiter tokens, while 13 found that outlier activations in ViTs tend to emerge in oow-information tokens.
better understand the nature of massive activations, we analyze their spatial variance by measuring the mean and variance of feature values across image patches. Feature dimensions are ranked by their averaged activation magnitude across patches. We then compare the top-ranked dimension (massive activations) with the 10th and 20th highest-ranked mon-massive) dimensions to assess their spatial discriminability. As shown in Table [ the variance of the massive activations (Ist) is substantially lower relative its mean compared to the non massive activations (10th and 20th), indicating that massive activations are less spatially discriminative. This suggests that massive activations encode limited local information.
Massive activations hold little local information.
AdaLN enables accurate localization of massive activations. It can be seen that the original features DiTs are computed 2 zt + Ag via residual connections, where a scaling factor a is applied to weight the block output A& (2.²) (see Equation ( To examine the potential link between massive activations and the residual scaling factor agz we analyze their dimensional correspondence in Figure and observe that they consistently co-occur in the same channels. Notably, og is regressed by the Adal.N-ze layer, suggesting that Adal NN-zer enables accurate localization of massive activations.
4.2 Channel-wise Modulation with AdaLN
Based on these observations, we hypothesize that AdaLN aggregates the activations in the block output into the dimension corresponding to the highest values in the scaling factor Ogs, resulting in dimennsion-comcenttaaedd massive activations. While such concentration may not be inherently
5

---

Input image
Pre-AdaLN
Post-AdaLN
Pre-AdaLN
Post-AdaLN
303-5
Flax
Figure AdaLN reduces massive activations and enhances feature semantics and discrimination. We visualize the activation magnitudes and feature maps of both pre-AdaLN and post-AdaLN features DiT block.
problematic, we argue that it is suboptimal for representation learning. Therefore, we further investigate how AdaLN interacts with the original features and massive activations within them.
AdaLN alleviates massive activations. To better understand how AdaLN interacts with massive activations, we compare the properties of pre-AdaLM and post-AdaLN intermediate features within DiT blocks. As shown Figure massive activations in feature channels are significantly reduced after AdaLN processing. This suggests that AdaLN can effectively identify such concentrated activations and adaptively normalize them through channel- modulation, using the scaling and shifting parameters.
AdaLN enhances feature spatial semantics and discrimination. Furthermore, visualize different feature maps in Figure It can be observed that the pre-AdaL feature maps DiTs exhibit limited semantic coherence, primarily capturing low-level textures such object color (e.g.. the cow and bird), and show weak discrimination between objects and background. contrast, the Post-AdaLN features display markedly improved spatial coherence and clearer semantic boundaries across object parts, suggesting that AdaLN improves spatial semantics and feature discrimination. More visualization results can found in the Appendix, where the same phenomenon is observed.
4.3 Extracting features from Diffusion Transformers
Based on these observations, we propose to extract DiT features with the channel-wise modulation by AdaLN. Specifically, the feature extraction process in DiTs be decomposed into stages: (1) extracting the original feature 2 from the DIT block Ak. and (2) modulating via adaptive channel-wise scaling and shifting using the AdaLN layer.
* (x Tk. BK MTPx((,ee) ボ = (1+ + 7k) LayerNorm (=+) + Ax
(8)
(9)
where af is the intermediate Pre-AdaLN feature, omitting the index i for clarify. 2t denotes the final extracted feature. For simplicity. we refer to it as 8 in the following section.
Channel discard. As shown in Figure 5 it can be observed that the Post-AdaLN features still exhibit few weakly massive activations, which will slightly compromise the local semantics of feature maps. Therefore, we propose a simple yet effective channel discard strategy to further suppress their influence. Specifically, we zero out the corresponding dimensions of these weakly massive activations in the post-AdaLN features as they hold little local information.
5
Experiments
5.1 Experimental Setup
Model Variants. We evaluate our strategies on four different pre-trained DiTs: Pixart-alpha (2]]. SD3 22). SD3-5 [22], and Flux (3). The channel-wise modulation with AdaLN operation for DìTs is conditioned on the input timestep t and text embedding except for Pixart-alpha 211. which

---

Table 2: Performance comparison on dataset SPair-71k. We present per-class and average PCK @0.10 10 on the test split. The compared methods are divided into two categories: supervised (S) and unsupervised (U). We report PCK per point results for the (U) methods and PCK per image results for the (S) methods, following [43 t: Integrating DINOv2 features for correspondence. The highest mAPs are highlighted in bold, while the second highest mAPs are underlined.
Method
Aer
Bike
Bird
Bost
Bottle
Bus
Car
Cat
Chair
Cow
Dog
Horse
Motor
Person
Plant
Sheep
Train
TV
V All
S SCOT 551
34,9
20.7
63.8
21.1
43.5
27.3
21.3
63.1
20.0
42.9
42.5
31.1
29.8
35.0
27.7
24.4
48.4
40,8
35.6
PMNC
35.9
74,9
36.5
42.1
48.8
40.0
21.1
58.1
50.5
40.1
54.1
43.3
59.9
SCouSAN 28
57.1
40.3
78.3
38.1
518
57.8
47.1
67.9
25
71.3
63.9
49.3
45.3
49.8
48.8
40.3
71.7
69.7
55.3
CAT 29
60.6
46.9
82.5
41.6
56.8
64.9
50,4
72.8
29.2
75.8
65.4
62.5
50.9
56.1
54.8
48.2
80.9
74.9
59.8
DHF [4
74.0
61.0
87.2
40.7
47.8
70.0
74,4
80,9
38.5
76.1
60.9
66.8
66.6
70.3
58.0
54.3
87.4
60.3
64.9
SD-DINO (S) 14
81.2
66.9
91.6
61.4
57.4
85.3
83.1
90.8
54.5
88.5
75.1
80.2
71.9
77.9
60.7
68.9
92.4
65.8
74.6
U ASIC [6]]
57.9
25.2
68.1
24.7
35.4
28.4
10.9
54.8
21.6
45.0
47.2
39,9
26.2
48.8
14.5
24.5
49.0
24.6
36.9
DINOv2+NN
12
72.7
62.0
40.4
52.3
51.5
71.1
36.2
67.1
68.2
30.7
62.0
24.2
DIFT [N
63.5
54.5
80.8
34.5
46.2
52.7
48.3
77.7
39.0
76.0
54.9
61.3
53.3
46.0
57.8
57.1
71.1
63.4
$7.7
DETF sss SEEFFaa.ssourr (ours)
66.5
56.8
56.3
40.1
513
58.6
58.8
813
47.8
79.8
60.2
68.5
65.7
73.5
64.4
67.8
69.7
66.5
64.6
Diu (ours)
74.3
65.0
88.
63.2
60.7
60.7
84.9
42.4
82.8
68.4
72.1
70.9
74.2
62.1
72.6
66.0
60.3
67.1
SD-DINO [+]
73.0 64.1 86.4
40.7
52.9
55.0
53.8
78.6
45.5
77.3
64.7
09.7
63.3
69.2
58.4
67.6
06.2
53.5
64.0
GeoAware-SC SCT
43
78.0
66.4 90.2
44.5
60.1
66.6
60.8
82.7
53.2
82.3
69.5
75.1
66.1
71.7
58.9
71.6
83.8
55.5
09.6
(ours))
79.1
67.8 90.6
56.1
69.2
65.0
59.4
84.6
72.6
76.7
69.9
75.5
62.1
74.4
59.6
(our))
76.3 67.8 88.5
50.8
55.8
60.9
60.8
82.9
47.1
81.3
70.5
72.3
70.0
76.2
62.6
72.5
64.8
56.5
67.6
Table 3: Performance comparison on datasets SPair-71k, AP-10K, and PF-Pascal datasets at
different PCK levels. We report the performance of the AP-10K intra-species (I.S.), cross- pross-species
(C.S.), and cross-family (C.F.) test sets. We report the PCK per image results following [43]. f: Integrating DINOv2 features for correspondence. The highest mAPs are highlighted in bold, while the second highest mAPs are underlined.
Method
0.01
SPait-1k
0.D5
0.10
0.01
AP-JOK-LS,
0.05
0.10
0.01
AP-JOK-CS,
0.05
0,10
0.01
AP-JOK-C.F
0.05
0.10
0.05
PF-Pascal
0.10
0.15
S
SCorSSN
28
3.6
36.3
55.3
81.5
93.3
96.6
CAT&++ 20
59.8
84.9
DHF 74
8.7
50.2
64.9
8.0
45.8
62.7
6.8
42.4
60.0
5.0
32.7
47.8
78.0
90.4
94.1
SD-DINO (5)
14
9.6
57.7
74.6
9.9
57.0
77.0
8.8
53.9
74.0
6.9
46.2
65.8
80.9
93.6
96.9
U DINOv2-NN
12
6.3
38.4
53.9
6.4
41.0
60.9
5.3
37.0
57.3
4.4
29.4
47.4
63.0
79.2
85.1
52.9
34.8
50.3
30.8
3.7
22.4
35.0
66.0
DVTF DilFaxs.s(our)
7.9
48.1
61.2
7.6
50.2
63.88
6.6
48.0
60:2
5.8
33.5
-77.9
88.5
96.2
97.3
(iTFrrusours)
8.3
49.3
64.0
7.6
19.7
02.2
6.8
48.2
61.7
5.7
3-12
48.9
59.5
95.8
97.6
SD-DINO 14
7.9
44.7
59.9
7.6
43.5
61.9
6.4
39.7
59.3
3.2
30.8
48.3
71.3
85.8
90.6
GeoAware-SC
43
9.9
49.1
65.4
11.3
49.8
68.7
9.3
44.9
64.6
7.4
34.9
52.7
74.0
86.2
90.7
DilFass.stour))
11.8
53.1
67.4 4
16.7
54.4
71.8
13.1
52.0
69.4 4
9.8
41.4
56.7
89.7
96.
97. 8
(ours))
10.1
50.0
66.0
12.9
52.0
68.8
11.2
40.7
67.1
9.0
39.9
55.6
89.6
95.9
97.6
follows its AdaLN design and is conditioned only on timestep L. Additional experimental details, including DiT configurations and hyperparameter settings, are provided in Appendix C.
Datasets and Evaluation Metric. We evaluate our DiTF on semantic correspondence, geometric correspondence, and temporal correspondence. For semantic correspondence, we conduct experi- ments on three popular benchmarks: SPair-7kk [57], PF-Pascal [58]. AP-10K benchmark [43]. The AP-10K benchmark 433 is new large-scale semantic correspondence benchmark built AP-10K 5.9. It comprises 2.61 million/36,000 data pairs, and spans three settings: the main intra-species set, the cross-species set, and the cross-family set. Following [8 14] we adopt the percentage of correct key-points (PCK) metric. See Appendix for the results of two additional correspondence tasks.
5.2 Main results on semantic correspondence
To evaluate our model DiTF, we conducted two types of semantic correspondence experiments: in-category semantic correspondence on SPair-7Ik (57) and PF-Pascal (58), and cross-category semantic correspondence on AP-10K (3
In-category Semantic Correspondence We conducted comprehensive experiments across different datasets where the results can be found in Tables 2 and From these results, we could obtain the following observations: 1) State-of-the-art performance. Overall, our models achieve of the-art performance across different datasets. Specifically, our model DiTF (iTFsds-s(ours) achieves 2.6% f on Spair-71k and 2.6% ? on AP- 2) Superior feature extraction capabilities. It is readily apparent that our feature extraction models for DiTs, such as DÝTF (ourr) and (uTFriux(uur)) outperform traditional feature extractors like SD2-1 (DIFT (8) and DINOv2 [12] across various datasets, demonstrating their strong and robust feature extraction abilities. Specifically, our Dur (ours) achieves performance of 67.1 % (9.4% 1 over DIFT) on the Spair-71k dataset, and obtains performance of 62.2% (1.3% 1 over DINOv2) dataset AP-10K-1S 3) Feature
7

---

Drigita
70
70
70
70
50
40
40
60
20
20
20
10
Sp0-TZ APASSC AMINGES 45-66C7.
Sui-Te AP-AEES AN-RECS AN-W5CT.
ANINGCS ANAECEE
Sni-TX ANARGAS
ANIIECA ANINGGE
Oo Pinwtalphh
60 Flux
Figure 6: Investigation of the impact of channel modulation with AdaLN and channel discard strategy across different datasets.
Source Image
DINOv2
DIFT
SD+DINO
DiTF hos (ours)
Target Image
CANAIS
Figure 7: Visualization of semantic correspondence prediction SPair-71k using different features. Different colors represent different key points where circles denote correctly predicted points under the threshold atbor 0.1 and crosses denote incorrect matches.
redundancy across DiTs and DINOv2: As shown in Table 2 when integrating DINOv2, our DiTFaa3-s achieves notable performance increase (from 64.6% 72.2%) In comparison, our DiTF shows a limited improvement, only a 0.5% gain (from 67.1% to 67.6%) We interpret this performance discrepancy from feature aligmment. FLUX aligns closely with DINOv2, causing redundancy, while SD3.5 benefits more from DINOv2's 's complementary distribution. This alignment may arise from two factors: Improved alignment between diffusion models and DINOv2 with larger models and longer training 60); Overlap in training data.
Cross-category Semantic Correspondence In addition, conducted cross-category semantic correspondence experiments on the AP-10K dataset, as shown Table The results indicate that models enable the extraction of accurate semantic features for cross-category image pairs and the drawing of correct correspondences, outperforming the previous best method, Geaeware [43), by nearly 4.8% (64.6% to 69.4% oon AP-IOK-C.S. Furthermore, the results show that previous methods, such as DINOv2 and DIFT, experience significant performance degradation on the cross-category task. In contrast, our model demonstrates stronger robustness for the cross-category correspondence.
5.3 Further analysis
Evaluation of Channel Modulation with AdaLN. We conducted comprehensive semantic cor- respondence experiments across different datasets on various DiTs. The results, as depicted in Figure and T reveal several key observations: 1) Effectiveness. When integrated with the channel modulation by AdaLNN, the DiT models experience an absolute performance boost of more than 20%, demonstrating that AdaLN is capable of effectively mitigating massive activations and

---

Table 4: Ablation study of our model DiTFrrm: the dataset SPair-7Ik and AP-10K. We report the PCK per point for SPair-7Ik and PCK per image AP-IOK-LS.
Table S: Conditions ablation study of AdaLN for feature channel-wise modulation SPair-71k. We report PCK 00 per point.
SPair-7Ik
AP-IOK-LS.
Condition
Pixart-Alpha
SD3
SD3-5
Flux
Model Variants
0.01 0.05 0.10 0.01 0.05
0.10
original
35.6
24.8
39.8
29.9
Original
14.6
29.9
16.5
33.9
17.9
36.8
20.3
Adal.N
7.8
52.3
65.3
46.8
60.3
55.3
54.6
58.2
60.9
Channel discard
8.3
56.3
67.1
7.6
49.7
62.2
t &
54.9
58.3
63.6
Table 6: Semantic segmentation on ADE20K.
Method
mloU TM
mloU
self-supervised pre-training
DINOv2 [2
47.7
53.1
SD-based pre-training
VPD
53.7
54.4
DiTsebased pre-training
Figure 8: Visualization of multi-to-multi correspon- dences with our model. Green lines indicate correct matches and red incorrect
Di AdaLN)
43.8
44.8
DiTFriux
53.6
54.8
enhancing feature semantics and discrimination. 2) Robustness. These results indicate that channel modulation with AdaLN enables the accurate normalization of features across different categories, leading to significant improvements on cross-category datasets.
Evaluation of Channel Discard strategy. As shown in Figure Table 4 our channel discard strategy can effectively locate and eliminate the weakly massive activations in the DiT feature across different DiTs, leading smoother feature maps and higher correspondence matching accuracy.
We can observe that
Impact of conditions in AdaLN for Channel-wise Modulation. From the results in Table 5 we can observe that the timestep is essential for modulating massive activations, bringing significant performance boost. In contrast, attempting to modulate features solely based on e proves ineffective. This phenomenon arises from the inherent nature of DiTs. During training. DiTs introduce noise to images at varying levels of I, making the timestep essential condition for feature modulation. It serves as key factor in locating the channel positions of massive activations and enhancing feature quality, enabling their transformations into a clean and semantically meaningful representation.
Qualitative Results. We visualize some semantic correspondence predictions from different models in Figures 7 and g The visualization reveals that our DiT-based model yields robust and accurate correspondences across various complex scenes, including changes in viewpoint, multiple objects, and dense matching. These results demonstrate the superiority and robustness of our models.
5.4 Other Task: Semantic Segmentation
To assess the generalizability of DiTF, we conduct semantic segmentation experiments ADE20K [61]. We extract activated DiT features using DiTF without applying the channel dis- card strategy and train segmentation head following DINOv2 [12). As shown in Table DiTF outperforms DINOv2, with channel modulation of AdaLN significantly enhancing feature semantics and discriminability, leading substantial mloU improvement (from 44.8 to 54.8). These results highlight the effectiveness and broad applicability of our approach.
6 Conclusion
In this paper, we identify and characterize massive activations in Diffusion Transformers (DiTs). We observe that these activations consistently emerge in few fixed dimensions across all image patch tokens and carry limited local information. We further demonstrate that the built-in AdaLN mechanism in DiTs effectively suppresses massive activations while enhancing feature semantics and discrininability. Building on these insights, we propose training-free N-aaed framework, Diffusion Transformer Feature (DiTF), which extracts semantically discriminative features from DiTs through channel-wise modulation using AdaLN. Extensive experiments on visual correspondence tasks validate the robustness and effectiveness of our approach.

---

7 Limitations and Future Work.
In this work, we primarily focus on understanding and mitigating massive activations in Diffusion Transformers (DiTs) from the perspective of representation learning. However, the emergence and potential roles of massive activations in the generative process of DiTs remain underexplored. We believe that investigating these activations from a generative viewpoint could provide deeper insights into their underlying mechanisms and potentially contribute improving generative performance.
Acknowledgements
The paper is supported in part by the National Natural Science Foundation of China (No. U21B2013, 62325109), and in part by the Shanghai 'The Belt and Road' Young Scholar Exchange Grant (24510742000). Mehrtash Harandi is supported by the Australian Research Council (ARC) Discovery Program DP250100262.
10

---

References
[1] Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Unsupervised part segmentation through disentangling appearance and shape. In CVPR, 2021.
[2] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences, 4rX14:2203.08414 2022.
[3] Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with shot using all-purpose feature matching. arXiv:2305.133: 13310, 2023.
[4] Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamameh. Slime: Segment like me. (tt.:2309.031799 2023.
[5] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen-Or. Cross- image attention for zero-shot appearance transfer. In SIGGRAPH, 2024.
[6] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-based image translation. In CVPR, 2020.
[7] Sooyeon Go, Kyungmook Choi, Minjung Shin, and Youngjung Uh. Eye-for-an-eye: Appearance transfer with semantic correspondence in diffusion models. arXiv:2406. 0700000000000 2024.
[8] Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Towards grand unification of object tracking. In ECCV, 2022.
[9] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better to track together. arXiv:2307:07635,, 2023.
[10] Qianqian Wang. Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In ICCV, 2023.
[11] Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised point tracking in single video. In ECCV, 2024.
[12] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193 2023.
[13] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv:2309. 16588, 2023.
[14] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Hiuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. NeurlPS, 2024.
[15] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[16] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurlPS, 2022.
[17) Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasac- chi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. NeurlPS, 2024.
[18] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 36:1363-1389 2023.
[19] Xinghui Li, Jingyi Lu, Kai Han, and Victor Adrian Prisacariu. Sd4match: Learning to prompt stable diffusion model for semantic matching, In CVPR, 2024.

---

[20] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models, arXiv preprint 11X112240..1766, 2024.
[21] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv:2310.00426, 2023.
[22] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICML, 2024.
[23] black-forest
labs.
Flux.
https://blackkoorettieee
announcing-blacce blrebbllec-------aattea---- 2024.
[24]
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. In European Conference on Computer Vision, pages 143-160. Springer, 2024.
[25] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
[26] David G Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
[27] Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N Sinha. Patchmatch-based neigh- borhood consensus for semantic correspondence. In CVPR, 2021.
[28] Shuaiyi Huang, Luyu Yang, Bo He, Songyang Zhang, Xuming He, and Abhinav Shrivastava. Learning semantic correspondence with sparse annotations. ECCV, 2022.
[29] Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. TPAMI, 2022.
[30] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. NeurIPS, 2021.
[31] Seungwook Kim, Juhong Min, and Minsu Cho. Transformatcher: Match-to-match attention for semantic correspondence. In CVPR, 2022.
[32] Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[33] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020.
[34] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. NeurIPS, 2024.
[35] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv:2112.05814, 2021.
[36] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[37] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NeurlPS, 2021.
[38] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv:21 2021.
[39] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. 12.0::21/..00330 2021.
[40] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. 21X 2021.
12

---

[41] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
[42] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In ICCV, 2023.
[43] Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. Telling left from right: Identifying geonetry-aware semantic correspondence. In CVPR, 2024.
[44] Jun Zhao, Zhihao Zhang, Yide Ma, Qi Zhang, Tao Gui, Luhui Gao, and Xuanjing Huang. Unveiling a core linguistic region in large language models. arXiv preprint 1rX1v22310.14928, 2023.
[45] Wujiang Xu, Qitian Wu, Zujie Liang, Jiaojiao Han, Xuying Ning, Yunxiao Shi, Wenfang Lin, and Yongfeng Zhang, SLMRec: Distilling large language models into small for sequential recommendation. In The Thirteenth International Conference on Learning Representations, 2025.
[46] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
[47] Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du Zirui Liu, and Yongieng Zhang. Massive values in self-attention modules are the key contextual knowledge understanding. arXiv preprint arXiv:2502.01563, 2025.
[48] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
[49] Jiawei Yang, Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision transformers. In European Conference on Computer Vision, pages 453-469. Springer, 2024.
[50] Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 181 18144-18154, 2025.
[51] Tianchen Zhao, Tongcheng Fang. Haofeng Huang, Rui Wan, Widyadewi Soedarmadji, Enshu Liu, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. In The Thirteenth International Conference on Learning Representations, 2025.
[52] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.0640, 2016.
[53] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[55] Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In CVPR, 2020.
[56] Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. In ICCV, 2023.
[57] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arXiv: 10543, 2019.
[58] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In CVPR, 2016.
13

---

[59] Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. 1.Xi:22088.1617 2021.
[60] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940. 2024.
[61] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[62] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. NeurlPS, 2018.
[63] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR workshops, 2018.
[64] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Mare Pollefeys, Josef Sivie, Akihiko Torii, and Torsten Sattler. D2-net: A trainable cnn for joint detection and description of local features.
arXiv:1905.03561, 2019.
[65] Michat Tyszkiewicz. Pascal Fua, and Eduard Trulls. Disk: Learning local features with policy gradient. NeurIPS, 2020.
[66] Zixin Luo, Tianwei Shen, Lei Zhou, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. Contextdesc: Local descriptor augmentation with cross-modality context. In CVPR, 2019.
[67] Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. NeurIPS, 2019.
[68] Qianqian Wang. Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. In ECCV, 2020.
[69] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
[70] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint wwXiv:1809.0322. 2018.
[71] Yingdong Hu, Renhao Wang, Kaifeng Zhang, and Yang Gao. Semantic-aware fine-grained correspondence. In European Conference on Computer Vision, pages 97-115. Springer, 2022.
[72] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[73] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-97388 2020.
[74] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[75] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
[76] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/VVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
14

---

[77] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 25294, 2022.
[78] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 5173-5182, 2017.
[79] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge video object segmentation. arXiv preprini arXiv:/704.0067, 2017.
[80] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuse Yang. Joint-task self superviee learning for temporal correspondence. Advances in Neural Informa- fion Processing Systems, 32, 2019.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions and scope are summarized in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations and future works are discussed in the main body.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We have already provided the proof of theoretical results in the experiment
section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Datasets and codes are available for double-blind review in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Datasets and codes are available for double-blind review in the supplementary
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/tips.cc/ppbliii wwtppbblcggguiiieecceeeeepooeeee http://ddese/eee/e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are fully provided in the main manuscript and the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the average results of open-source models under three random seeds (2024, 2025, 2026). Besides, the average performance is taken from three experimental runs for all proprietary models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We extract representation from DiTs on one L40s GPU for about 10 minutes, max GPU memory is about 16 GB.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/ http://p....ppullectccccccccleed
Answer: [Yes]
Justification: We conducted the research in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The scope of our research does not involve data or models with a high risk for misuse. We utilize publicly available, non-sensitive datasets and models that do require special safeguards for responsible release.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We strictly adhered the license terms for all the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This question does not apply research new assets introduced. Our study solely relies on existing, publicly available resources, and thus, no additional documentation for new assets is necessary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This question does not apply to our research as it does not involve crowdsoure- ing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This question is not applicable to our research as no human subjects were involved.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
22

---

Appendix
A
Diffusion Transformer Architecture
Noise
Scale
Feodtorward
Decoder
Stale, Shift
Layer Norm
x
DiT Block A,
Scale
Salt-
Attention
Encoder
Stale, Shift
Layer Norm
MLP
Input Feature
Diffusion Transformer
DiT Block with AdaLN
Figure 9: DiT architecture illustration.
We strictly follow the architecture used DìT (36] We provide an illustration of the DiT architecture shown in Figure
B
More Visualization Results
B.1 Massive Activations DiTs
To further illustrate the emergence of massive activations in DiT', we provide additional visualization results in Figure 13 We show the eyverroormnoommliiae activation magnitudes original features from SD2-1 and various DiTs. While SD2-1 exhibits smooth activations, DiTs consistently show spikes concentrated in a few fixed dimensions across all patch tokens, revealing a fundamental difference that contributes to their degraded performance.
Original feature 27
Residual spali Ci)
Original Kature
Residual Oa
Pixart-Alpha
SD3
Original Scature
Residual scale
Original fcature
Residual scale
SD3-5
Flux
Figure 10: Massive activations dimensions align with the residual scaling factor og- We visualize the magnitudes for the original feature 2 ++1 and residual scaling factor ag.
B.2 Massive Activations Dimensions Align with Residual Scaling Factor
In this section, we provide additional visualizations to examine the dimensional alignment between massive activations and the residual scaling factor a from the AdaLN layer. As illustrated in Fig-
23

---

massive activations consistently co-occur with large values of Cr in the same dimensions across all DiTs.
ure I0,
B.3
Channel-wise Modulation with AdaLN
To comprehensively demonstrate the impact of the built-in AdaLN DiTs, we provide additional visualizations comparing Pre-AdaLN and post-Adala features across various models, including SD3-5 (Figure IT). Pixart-Alpha (Figure [4), SD3 (Figure (3), and Flux (Figure (6). These re- sults consistently show that AdaLN accurately localizes and normalizes massive activations, while enhancing feature semantics and discrimination through effective channel-wise modulation.
Inpur image
Pre-AdaLNN
Post-Adal.N
Pre-AdaL
Post-AdaL.N
Figure 11: Comparisons of pre-AdaLN and post-AdaLN features in SD3-5.
C Further Implementation Details
Configurations of different DiTs. We employ the pre-trained Diffusion Transformers (DiTs) as feature extractor for semantic correspondence. Formally, we decompose the universal feature extraction process in DiTs into two stages: (1) extracting the original feature 2f from the DiT block
Ag. and (2) modulating via adaptive channel-wis scaling and shifting using the AdaLN layer. For
Pixart-alpha (21). SD3 [22]. and SD3-5
(22), we first extract the Pre-AdaLN feature z (l.22 and then
activate it as follows.
2
(k.2) Ax(2t),
mk. S (t, c)
(10)
24 -1+1 LayerNorm
(2(k.22)
(k.2)
(11)
As some of the Flux model's blocks contain only one group of Adal. layer, we extract pre-AdaLN feature 2; (k.1) and then activate it as follows.
2t (k,1) = Ac(zt), The St = MLPP((t()
(12)
* (1 LayerNorm (+ (((:1)) 1
(13)
24

---

70
60
60
50
40
30
10
11 16
46 51 56
20 140 260 350
620 740 860 980 Timestep
Block index k
Figure 12: Investigation of the DiT block index & and timestep We report the results DiTFrius on dataset Spair-7lk.
Table 7: Configurations of different DiTs for semantic correspondence.
Method
Layers
Hidden size
Timestep
Block index
DiTF pixartt
28
1152
141
DiTFspa
24
1536
340
DiTFs
38
2432
380
23
DiTFriux
3072
260
28
The configurations of different DiT's can be found Table where the total time step is 1000. We set the input image size 960x960 for DiTs and 840x840 for the model DINOv2.
Investigation on block index & and timestep The layer index & and the timestep f are critical hyperparameters that influence the quality of the extracted features from Diffusion Transformers (DiTs). Previous studies [8] [4] have conducted thorough investigations to identify the optimal layer index and timestep for Stable Diffusion. To explore the effects of varying & and t in DiTs, we conducted grid search experiments to identify the optimal hyperparameters. The results are presented in Figure 12 The figure reveals that features extracted from the middle layer achieve optimal performance across different DiTs. Furthermore, feature extraction in DiTs is robust to the timestep for semantic correspondence tasks, wide range of achieves excellent performance.
Integration DINOv2 feature. We extracted DINOv2 features from the token facet of the 11th layer of the model. We then concatenated the DiT's features with the DINOv2 features in the channel dimension. To improve the efficiency of correspondence calculation, we computed Principal Component Analysis (PCA) across the pair of images for the features extracted from DiT, follows:
PCA (3*1)=))
(14)
where 2* st are the extracted source and target image DiT features. We only apply the PCA operation to SD3-5 and Flux features due to their high dimension and set the output dimension size as 1280.
D Geometric Correspondence
To comprehensively evaluate our model DiTF, we conduct additional experiments on geometric correspondence.
Datasets. Following [18] evaluate model the HPatches benchmark (78), which comprises 116 sequences: 57 with illumination changes and 59 with viewpoint variations. Adopting the approach from CAPS (68) detect up 1,000 key points per image and apply ev2.find w.2.findHomographhy to estimate homography using mutual nearest neighbor matches.
Metric. We adopt the corner correctness metric for evaluation where compute the average error between the four estimated corners of one image and the ground-truth corners with a threshold e pixels, following [68 [8]..
Results. To comprehensively evaluate our models, we conducted geometric correspondence ex- periments on the HPatches benchmark (78). as detailed in Table g From the results, it can be observed that our model enables robust feature extraction for image pairs and draws precise geo- metric correspondence. Specifically, our model DiTFy achieve comparable performance 41.9% compared state-of-the-ar model DIFT (SD-based). These results show that Diffusion Transformers be employed an effective feature extractor for geometric correspondence.
25

---

Table 8: Geometric correspondence results dataset HPatches. We report the homography estimation accuracy [%] at 1, 3, 5 pixels.
Geometric
All
Viewpoint Change
Illumination Change
Method
Supervision
3
3
SIFT
None
40.2
79.3
55.4
72.1
54.6
81.5
L.F-Net 62
34.4
62.2
73.7
16.3
43.9
60.7
53.5
81.9
87.7
SuperPoint
36.4
72.7
82.6
56.1
$1.9
90.8
98.1
D2-Net
16.7
61.0
75.9
38.0
30.2
84.9
Strong
DISK 66
40.2
70.6
81.5
51.4
67.9
58.5
91.2
96.2
ContexDDesc
66
40.9
73.0
82.2
29.6
60.7
72.5
53.1
86.2
92.7
R2D2 667
40.0
74.4
84.3
60.4
73.9
54.6
95.4
w/ SuperPoint kp.
CAPS 68
Weak
44.8
76.3
85.2
35.7
62.9
74.3
$4.6
90.8
96.9
DINO 12
38.9
70.0
81.7
21.4
50.7
67.1
57.7
90.8
97.3
OpenCIIP
33.3
67.2
78.0
18.6
45.0
59.6
49.2
97.7
None
DIFT 08
45.6
83.1
30.4
56.8
61.9
41.9
70.7
79.5
22.0
50.8
63.4
62.5
91.3
96.2
Table 9: Tempral correspondence results on DAVIS-2017. We report the region-based similarity J and contour-based accuracy F for DAVIS. Pre-: Pre-trained on videos.
DAVIS
Pre-
Method
Dataset
J&F
Jm
F m
MAST
65.5
63.3
67.6
YT-VOS 70
SFC 771
71.2
68.3
74.0
InstDis 77
66.4
63.9
68.9
MoCo 73
65.9
63.4
68.4
SimCLR 74
66.9
64.4
69.4
ImageNet 72
BYOL 75
66.5
64.0
69.0
w/o labels
SimSiam 76
67.2
64.8
68.8
DINO 32
71.4
67.9
74.9
OpenCLIP 66
62.5
60.6
64.4
LAION
DIFT 18
70.0
67.4
72.5
DiTF
72.2
69.2
75.1
E
Temporal Correspondence
In addition, conduct experiments to verify the temporal correspondence capability of our DiTF. Specifically, we investigate DiTF's performance on video object segmentation and pose tracking tasks, employing DiT's feature extractor for correspondence.
Datasets. We conduct experiments on the challenge video dataset: DAVIS-2017 video instance segmentation benchmark 77), following [18).
Metric. Following 18 80
adopt the region- similarity I and contour-based accuracy FF as the performance metric where we segment the nearest neighbors between the consecutive video frames based on the representation similarity.
Results. The temporal correspondence results can be found in Table From the results, it can be observed that our model exhibits superior capability extract video frame features for temporal correspondence. Specifically, our model achieves 72.2% on the dataset DAVIS, which outperforms the previous state-of-the-art DIFT by 2.2%, demonstrating the superior effectiveness of model.
F Qualitative Results on AP-10K
We show the qualitative comparison of our Diffusion Transformer model DiTF with both Stable Diffusion (SD) and SD+DINO [14 in AP-10K intra-species (Figure 17). cross-species (Figure T8] and cross-family (Fiurre 19) subset.
26

---

Inper image
Models
Pixart-Alpha
SD3
SD3-5
Prod
Flux
Figure 13: Massive activations in DiTs. SDv2-1 does not suffer from massive activations. However, all DiTs exhibit massive activations, which concentrate on very few fixed dimensions across all image patch tokens.
27

---

Input
Pre-AdaLN
Post-AdaLN
Pre-AdaLN
Post-AdaLN
Figure 14: Comparisons of pre-AdaLN and post-AdaLN features in Pixart-Alpha.
Input image
Pre-AdaLN
Post-AdaLN
Pre-AdaLN
Post-AdaLN
Figure 15: Comparisons of pre-AdaLNN and Post-AdaLN features in SD3.
28

---

Input
Pre-AdaLNN
Post-AdaLNN
Pre-AdaLN
Post-AdaLL
Figure 16: Comparisons of pre-AdaLN and Post-AdaLN features in Flux.
29

---

SD
SD+DINO
DITE Bix (ours)
Figure 17: Qualitative comparison on the AP-10K intra-species set. Different colors represent different key points where circles denote correctly predicted points and crosses denote for incorrect matches.
30

---

SD
SD+DINO
DITF
Figure 18: Qualitative comparison the AP-10K cross-species set.
SD+DINO Figure 19: Qualitative comparison on the AP-10K cross-family set.
SD
DITF
31


Reviews:
[{"review_id": "7T9sMyBqsq", "reviewer": "Reviewer_ntGm", "summary": "This paper investigates the massive activation phenomenon in Diffusion Transformers (DiTs) for visual correspondence tasks. The authors identify that DiTs exhibit massive activations concentrated in specific feature dimensions across all image patch tokens, which leads to uninformative representations and performance degradation. They trace this phenomenon to the AdaLN-zero layers and propose DiTF (Diffusion Transformer Feature), a training-free framework that leverages AdaLN's channel-wise modulation to suppress massive activations and extract semantically discriminative features. The method achieves state-of-the-art performance on multiple correspondence benchmarks, with significant improvements over existing approaches.", "questions": "1. **Theoretical Foundation**: Can you provide theoretical insights into why massive activations appear uniformly across all image patch tokens in DiTs, unlike the token-specific patterns in LLMs? What architectural or training differences in DiTs might explain this phenomenon?\n2. **Computational Efficiency**: Could you provide a detailed computational cost comparison between your billion-parameter DiT models and much smaller supervised baselines? How do you justify the computational overhead given the dramatic parameter size differences?\n3. **Hyperparameter Sensitivity**: How sensitive is the method to the channel discard threshold and other hyperparameters? Can you provide ablation studies showing performance across different threshold values and discuss the method's robustness?", "limitations": "The authors should more transparently discuss several limitations in the \"main\" paper (not only in supplementary): (1) the lack of theoretical understanding for the unique spatial distribution of massive activations in DiTs, (2) the significant computational overhead compared to smaller supervised methods, (3) the need for hyperparameter tuning and its robustness implications, and (4) the method's reliance on pre-trained models which may limit applicability to novel architectures or training paradigms.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. **Novel Discovery and Analysis**: This work provides the first systematic analysis of massive activations in DiTs, identifying unique characteristics distinct from LLMs (concentration across all spatial locations vs. specific tokens). The connection between massive activations and AdaLN-zero residual scaling factors is insightful.\n2. **Practical Training-Free Solution**: The proposed DiTF framework is elegant and immediately applicable to existing pre-trained DiTs without requiring additional training, making it highly practical for the community.\n3. **Comprehensive Experimental Validation**: The authors demonstrate consistent improvements across multiple DiT architectures (Pixart-alpha, SD3, SD3-5, Flux) and datasets (SPair-71k, AP-10K, PF-Pascal), establishing strong empirical evidence.\n4. **Strong Performance Gains**: The method achieves substantial improvements (+9.4% on SPair-71k, +4.4% on AP-10K-C.S.) and establishes new state-of-the-art results for DiT-based visual correspondence.\n\n**Weaknesses:**\n\n1. **Insufficient Theoretical Analysis**: While the paper identifies the connection between massive activations and AdaLN-zero, it lacks theoretical explanation for why DiTs exhibit massive activations across *all* image patch tokens, unlike the token-specific patterns observed in LLMs. This fundamental difference deserves deeper theoretical investigation.\n2. **Inadequate Computational Cost Analysis**: The paper compares against models with vastly different computational requirements without transparent discussion. For instance, Flux operates at billion-parameter scale while supervised methods like CATs++ use ~50M parameters. While supervised methods require labeled data training, their dramatically smaller size represents a significant efficiency advantage that should be explicitly acknowledged and discussed.\n3. **Limited Hyperparameter Analysis**: The paper introduces several hyperparameters (channel discard thresholds, condition dependencies) but provides insufficient sensitivity analysis. The robustness of the method across different threshold values and its stability across diverse scenarios remain unclear.\n4. **Missing Related Work**: The paper fails to cite \"TinyFusion: Diffusion Transformers Learned Shallow\" (Fang et al., 2024), which also identified massive activation issues in DiTs and proposed Masked RepKD as a solution. This is a critical omission as it represents the first work to address massive activations in DiTs, even if at a preliminary level.\n5. **Lack of Transparency in Limitations**: While the supplementary material includes a limitations section, the main paper does not transparently discuss these constraints, which is important for reader understanding and future research directions.", "was_revised": true, "final_justification": "I hope that the revisions mentioned by the authors will be made as requested, and I will maintain my score as it is.", "rebuttal": "Thanks for your valuable suggestions. We’re glad that you found our work insightful and interesting. Please see the responses to your comments.\n\n### **Q1. More theoretical insights into the unique characteristics of massive activations in DiTs**\n\nThank you for your insightful comment. We would like to emphasize that our primary focus in this work is to investigate the impact of massive activations from a representation learning perspective. To inspire further research on visual generation, we provide a preliminary theoretical claim to offer insights into the underlying cause of this phenomenon.\n\n**Distinct training paradigms and supervisory signals result in different spatial locations of massive activations in LLMs and DiTs.**\n- LLMs are typically trained using an **autoregressive next-token prediction** objective, where each token prediction is conditioned on all preceding tokens. As the model generates the entire sequence token-by-token, the early tokens play a **disproportionately important role** in shaping the final output distribution. Consequently, the initial tokens (particularly starting token) tend to accumulate stronger activations, resulting in token-specific massive activations [1].\n- In contrast, Diffusion Transformers (DiTs) are trained under a **denoising diffusion** paradigm, where supervision is applied uniformly across all image patches. Each patch is **treated equivalently** during training, and the model learns to reconstruct all patches simultaneously. This **uniform treatment** results in a spatially uniform distribution of massive activations across tokens.\n- To verify our hypothesis, we conduct a comparative analysis of massive activations in two large language models **LLaMA3-8B** [4] and **LLaDA-8B** [5]. Notably, these two models share a similar architecture, and the primary difference lies in their training paradigms: LLaMA3-8B follows **next-token prediction** objective, while LLaDA-8B is trained using **masked diffusion modeling**. As shown in Table i, massive activations in LLaMA3-8B are concentrated in a **specific token**: the starting token (
). In contrast, LLaDA-8B exhibits massive activations in dimension 3848 **across all text tokens**. This distinction aligns with our theoretical claim that the choice of training paradigm fundamentally influences the activation distribution patterns.\n\n\n| Text tokens |<\\|startoftext\\|>|\"Summer\"|$~~$\"is\" |$~~$\"warm\"|$~~$\".\"|\"Winter\"|$~~$\"is\"|$~~$\"cold\"|$~~$\".\"|\n| :--: | :--: |:--: |:--: |:--: |:--: |:--: |:--: |:--: |:--: |\n| LLaMA3-8B (788) | 226.0|2.0|1.4|0.9|1.2|0.2|1.4|0.7|2.3|\n| LLaDA-8B (3848)|134.0|112.5|101.0|103.5|108.5|116.5|80.0|119.5|92.5|\n\n*Table i. Distribution of activation values along the massive activation dimension in LLaMA3-8B and LLaDA-8B. The dimension associated with massive activations is indicated by (·). The input sentence is: \"Summer is warm. Winter is cold.\".*\n\n### **Q2. Computational cost analysis**\n\nTo provide a more transparent comparison, we have included a detailed computational cost analysis in Table ii. All results are measured on a single NVIDIA L40S GPU (48 GB). Below, we elaborate on several key aspects relevant to this concern:\n\n**1) Training-free and zero-shot Inference:** \n\nDiTF operates in a fully training-free and zero-shot manner, leveraging pretrained Diffusion Transformers (DiTs) as universal feature extractors. This design enables direct application to a wide range of visual correspondence tasks **without any task-specific fine-tuning**. In contrast, supervised methods such as CATs++ [2] rely on costly and time-consuming task-specific training.\n\n**2) Efficient inference with simpler matching pipeline:** \n\nDiTF achieves faster inference despite using larger backbones, due to its **lightweight and direct matching pipeline**. Specifically, DiTF only requires a single forward pass to extract dense features from each image, followed by a **cosine similarity computation** for pixel-wise matching, where no trainable modules are introduced in the matching process. In contrast, supervised methods typically adopt a two-stage process:(1) Extract features via a CNN or Transformer backbone (2) Feed the feature pair into an **additional trainable cost aggregation network** to produce the final matching result. This additional network significantly **slows down inference**. As a result, although DiTF uses a large pretrained backbone, its inference time is **faster** than that of supervised as CATs++ [2] (**42.0 ms vs. 55.9 ms**), as shown in Table ii.\n\n**3) Negligible overhead from large-scale backbones:** \n\nThe feature extraction stage consumes **only a small portion of the overall runtime**. For instance, with the billion-parameter DiT (Flux) model, the feature extraction time is 6.21ms (42.0ms in total), accounting for **less than 15%** of total inference time. This suggests that the performance gain from better features outweighs the minor cost introduced by a larger backbone.\n\n| Model |Zero-shot|Inference Time \\[ms/pair\\] | Accuracy |\n| :-----: | :-------: |:-------: |:-------: |\n| CATs++| No |55.9(1.99)|59.8|\n| DINOv2 |Yes|21.0(2.53)|55.6|\n| DIFT|Yes|26.7(4.08)|57.7|\n| DiTF$_{SD3-5}$|Yes|29.5(4.90)|64.6|\n| DiTF$_{Flux}$|Yes|42.0(6.21)|67.1|\n\n*Table ii. Computational cost comparison of different models on the SPair-71k semantic correspondence benchmark. Inference time for feature extraction is denoted by (·)*\n\n### **Q3. Hyperparameter analysis**\n\n**Channel discard thresholds**: In our study, we set the default discard threshold to $m=10$, where a feature dimension is discarded (zeroed) if its mean activation magnitude **exceeds $m\\times$the global median**. To assess robustness, we evaluated thresholds $m\\in$ {14,12,10,8,6}. As Table iii shows, our method maintains stable performance across all thresholds and different DiT models. Notably, the normalized (weakened) massive activations remain ≥14× the global median, ensuring a clear distinction from normal activations. Such a wide margin ensures that varying the threshold $m$ (from 6 to 14) has minimal impact on performance, highlighting the robustness of our method.\n\n| Threshold  | $~$14 | $~$12 | $~$10 | $~~$8 | $~$6 |\n| :------------: | :--: | :--: | :--: | :--: | :--: |\n| Pixart-Alpha   | 58.5 | 58.5 | 58.5 | 58.5 | 58.5 |\n| SD3-5          | 64.6 | 64.6 | 64.6 | 64.6 | 64.6 |\n| Flux           | 67.1 | 67.1 | 67.1 | 67.1 | 66.5 |\n\n*Table iii. Performance with different channel discard thresholds on the SPair-71k semantic correspondence benchmark.*\n\n**Condition dependencies**: We have already included a detailed ablation study in Table 5, demonstrating that the condition timestep t plays a critical role in enabling AdaLN to modulate massive activations (Lines 248-254). More investigations of the block index k and the timestep t are provided in Figure 4 of the supplementary material.\n\n### **Q4. Related works**\nWe appreciate the suggestion and will cite TinyFusion [3] along with other related works in the revised version.\n\n### **Q5. Transparent discussion of limitations**\nIn the revised version, we will move the limitations and future work section from the supplementary material to the main paper, and include a discussion emphasizing the need for a deeper understanding of why such massive activations emerge during DiT training and what functional role they play in visual generation tasks.\n\n[1] Massive activations in large language models, COLM 2024.\n\n[2] CATs++: Boosting Cost Aggregation with Convolutions and Transformers, TPAMI 2022.\n\n[3] TinyFusion: Diffusion Transformers Learned Shallow, CVPR2025.\n\n[4] The llama 3 herd of models. \n\n[5] Large Language Diffusion Models."}, {"review_id": "RkI5JniLko", "reviewer": "Reviewer_f4nx", "summary": "The authors identify and analyze  “massive activation” in Diffusion Transformers (DiT): certain feature's activation exhibit significantly large activations that infected fine-grained tasks such as visual correspondence. To address this, they propose DiTF (Diffusion Transformer Feature), a modified DiT framework incorporating AdaLN, an adaptive LayerNorm that learns channel scaling and shifting parameters conditioned on the activation statistics. Also they discard the weakly massive activation after the AdaLN features by zero out the corresponding dimensions. Experiments on standard benchmarks demonstrate that DiTF substantially reduces activation peaks and variance, leading to around 5–10 % higher matching accuracy in semantic correspondence. This work is the first to diagnose massive \nactivations in DiT, offering a general normalization design that enhances stability and performance in large scale diffusion Transformer models", "questions": "1. In Section 4.2, it is stated that AdaLN alleviates massive activations. However, in Figure 5, after applying DiTF to the Flux model, the dimensions of the original massive activations and the normalized (weakened) massive activations appear to be inconsistent. Could you explain why this occurs, and whether it is reasonable?\n2. As mentioned in the weaknesses, could you provide additional comparisons of the SD3.5 and Flux  models on specific tasks before and after applying DiTF? I did not find these results in the paper’s Results section. Also, which version of the SD3.5 model are you using? (SD3.5 large or Medium)", "limitations": "This work lacks an analysis of its methodological limitations; the authors could examine potential constraints of their approach, including its impact on transferability to a broader set of tasks.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. This paper carefully evaluates the “massive activation” phenomenon in the DiT model and provides a detailed analysis of it. \n2. The exploration of the phenomenon and the application of the proposed method follow a logical flow, and the experimental results are relatively comprehensive.\n\nWeakness: \n1. The paper’s explanation of how AdaLN specifically mitigates the “massive activation” phenomenon is not sufficiently detailed. The evaluation metrics lack direct comparisons between the same base models (e.g., SD3.5 or Flux) with and without the DiTF framework", "was_revised": true, "final_justification": "The author addressed my concerns properly in their rebuttal. Based on the new discussion and explanation, I decided to raise my rating.", "rebuttal": "Thanks for your suggestions and for recognizing the novelty and contribution of our work. Please see the responses to your comments.\n\n### **Q1. Details of how AdaLN mitigates the massive activations**\n\nThank you for your valuable comments. We have described the channel-wise modulation strategy using AdaLN to address massive activations in Sections 4.2 and 4.3. To further clarify the mitigation process, we elaborate on the rationale and mechanism below.\n\n**1) Why utilize AdaLN to mitigate the massive activations?** \n\nIn DiTs, feature updates are computed via residual connections: $z_{t}^{k+1} = z_{t}^{k} + \\alpha_k A_k(z_{t}^{k}),$ where $\\alpha_k$ denotes the channel-wise residual scaling factor from the AdaLN-Zero layer. We observe that the massive activation dimensions in $z_{t}^{k+1}$ strongly align with large values of $\\alpha_k$ in the corresponding channels (as shown in Figure 4). This reveals an **inherent connection** between massive activations and the scaling behavior of AdaLN. Based on this insight, we leverage the adaptive nature of AdaLN to **identify and regulate** these massive activations.\n\n**2) How to mitigate the massive activations with AdaLN?**\n\nThe modulation process of AdaLN is formalized as:\n$$\n\\text{AdaLN}(z_{t}^{k}) = (1 + \\gamma_k) \\cdot \\text{LayerNorm}(z_{t}^{k}) + \\beta_k,\n$$ \nwhere $\\gamma_k \\in \\mathbb{R}^C$ and $\\beta_k \\in \\mathbb{R}^C$ are adaptive channel-wise scaling and shifting parameters, respectively. Applied to the DiT features $z_{t}^{k} \\in \\mathbb{R}^{C \\times H \\times W}$, this modulation allows AdaLN to **adaptively identify and suppress** dimensions with massive activation through the factor $(1 + \\gamma_k)$. Consequently, the massive activations are effectively mitigated by the channel-wise modulation of AdaLN (see Figure 5).\n\n### **Q2. Explanations of inconsistent dimensions in flux**\n\nThank you for your comment. We emphasize that AdaLN effectively suppresses the original massive activations at dimensions 154 and 1446 through channel-wise modulation. The observed inconsistency in the dimensions of (weakened) massive activations stems from the adaptive nature of AdaLN, which applies distinct scaling factors to each dimension based on learned statistics. While AdaLN strongly attenuates the dimensions corresponding to massive activations, it may slightly amplify others, resulting in a shift in the top activated dimensions after normalization. This behavior reflects the rebalancing effect of AdaLN and does not negatively impact downstream performance.\n\n\n### **Q3. Comparisons between models with and without the DiTF framework**\n\nWe have provided detailed direct comparisons between the same base models (e.g., SD3-5 or Flux) with and without the DiTF framework in Figure 6. For better readability, we summarized these results in Table i. The comparison clearly shows that incorporating DiTF leads to consistent and significant performance improvements across different DiT-based models.\n\n| Model| Pixart-alpha |SD3|SD3-5|Flux|\n| :--------: | :-------: |:-------: |:-------: |:-------: |\n| w/o DiTF|35.6|24.8|39.8|29.9|\n| w DiTF |58.5|58.0|64.6|67.1|\n\n*Table i. Comparisons between models with and without the DiTF on the SPair-71k semantic correspondence benchmark.*\n\nIn the paper, SD3 refers to the SD3-Medium model, while SD3-5 corresponds to the SD3.5-Large model. Additionally, we conducted experiments on SD3.5-Medium, where our DiTF led to a substantial performance improvement from **38.8 to 58.3**, demonstrating the effectiveness of our method. To improve clarity, we will provide more detailed descriptions of the different DiT variants in the final revision.\n\n### **Q4. Generalizability on various CV tasks**\n\nWe have conducted **semantic segmentation** experiments on ADE20K, as presented in Table 6 of the main paper. Specifically, our method significantly improves the performance of DiT features, increasing the mIoU **from 43.8 to 53.6** (*DiTF$_{Flux}$*). Additionally, we perform **linear probing** experiments on **ImageNet classification** using DiT features, following the same protocol as DINOv2 [1]. We use the globally pooled DiT features as input. Our method again yields a substantial improvement, boosting accuracy **from 65.8 to 72.6** (*DiTF$_{Flux}$*). These results demonstrate the effectiveness and general applicability of our approach across diverse vision tasks.\n\n### **Q5. Discussion of limitations and future works**\n\nWe have discussed the limitations and potential future directions in the supplementary material. In the final revision, we will expand upon these points and incorporate them into the main text to enhance clarity and completeness. In response to your suggestion, we will also apply color coding to Figures 1a and 2 for improved readability.\n\n\n[1] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2025."}, {"review_id": "abWa0GGfPJ", "reviewer": "Reviewer_92HZ", "summary": "This paper investigates the usage of pre-trained Diffusion Transformers (DiTs) for visual correspondence tasks and identifies a phenomenon of \"massive activations\" in DiT feature maps. To mitigate this, the authors propose a training-free feature extraction method (DiTF) that leverages existing AdaLN layers and introduces a simple channel discard strategy to suppress such activations. The paper demonstrates consistent improvements across several benchmarks.", "questions": "If the authors can address my main concerns in the rebuttal, I will adjust my rating accordingly:\n\n1. Perhaps I misunderstood, but could the authors clarify the distinction (if any) between the proposed AdaLN and the AdaLN that already exists in DiT models? As it stands, this appears to be reused rather than redesigned.\n\n2. Your work suppresses or even zeros out massive activations. However, the Massive Activation paper you cite argues that these activations are critical for model performance. Why do you observe the opposite? Could you elaborate on this contradiction?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\nThe identification and quantification of massive activations in DiTs is the first attempt in the community. Ablation studies and visualizations are massive. The performances are evaluated across several datasets with consistent gains. The reference is sufficient and comprehensive, and up-to-date.\n\nWeaknesses\n1. The main concern is that the core contribution is relatively limited and heavily dependent on prior works, which undermines the novelty and significance required for a NeurIPS-level submission. It's more like a reimplementation of [20] in DiT [36] without novel insights. The entire pipeline builds upon existing DiT architectures and the concept of massive activations. The only novel contribution appears to be the channel discard strategy, which, while useful, cannot reach the bar for NeurIPS.\n\n3. Misleading attribution of contribution: Several operations, particularly the use of AdaLN for normalization and modulation, are not proposed by this paper, but are rather built-in functionalities of DiTs. While authors claim that 'DiTF employs AdaLN to adaptively localize and normalize massive activations with channel-wise modulation,' which might be misleading.\n\n4. Lack of theoretical depth: Many conclusions are derived from visual inspection or statistical correlations without formal analysis or proof. There is no theoretical justification for why the proposed modifications improve semantic representation. E.g., Massive activations hold little local information; however, the claim is made based on the low variance of the massive activations on the feature, which cannot directly lead to the conclusion.\n\n5. Inconsistency with prior literature: According to the prior work [20] “Massive Activations in LLMs”, massive activations are correlated with important representational capacity, disabling them will lead to performance degradation. However, this paper argues for suppressing them entirely (even zeroing them out) to improve the performances, which appears contradictory and is not justified. It's strongly recommended to give a further explanation or analysis.\n\n6. Since the proposed method is a universal paradigm, not specified for the task, it would be beneficial to also test it on some common CV tasks, like image classification.\n\n\nMinors: The manuscript would benefit from a thorough proofreading—e.g., in line 107, “AdaIN” appears to be a typo and should be corrected. [45] seems to be a wrong citation where Massive activations were not discussed. Incorrect assumptions: The paper simplifies AdaLN-zero to AdaLN, ignoring the important differences in their initialization schemes. This is conceptually incorrect.\n\nFinal Justification\nWhile the experimental results are decent, the paper relies too much on prior architectural elements and empirical heuristics. The channel discard strategy, being the only new technical contribution, is not substantial enough to warrant acceptance at NeurIPS. The work would benefit significantly from deeper theoretical insights or architectural innovations rather than retrofitting observations on existing models.", "was_revised": true, "final_justification": "Please see my final response", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our contributions on massive activations in DiTs and address your concerns as follows:\n\n### **Q1. Clarification of core contributions and novelty**\nWe would like to clarify that our method is **fundamentally different** from a simple reimplementation of [1] within the DiTs, and it introduces novel insights and contributions beyond them. In particular, we emphasize the following key aspects:\n\n**1) Unique characteristics and insightful analysis of massive activations in DiTs.**\n\n- Compared to [1], we identify **distinctive characteristics** of massive activations in DiTs, which **differ fundamentally** from those observed in large language models (LLMs) [1]. In DiTs, the massive activations are consistently concentrated in a few fixed dimensions **across all spatial tokens** (see Figures 2 and 3). In contrast, massive activations in LLMs typically occur in **specific input tokens** (e.g., starting token).\n- Furthermore, our analysis reveals that the unique characteristic of massive activations in DiTs is closely associated with the **channel-wise residual scaling factors** introduced by AdaLN-Zero. Specifically, DiT feature updates are computed via residual connections: $z_{t}^{k+1} = z_{t}^{k} + \\alpha_k A_k(z_{t}^{k}),$ where we observe a strong alignment between the dimensions exhibiting massive activations and those corresponding to large values of the residual scaling factors $\\alpha_k$ (see Figure 4 and supplementary material). Such theoretical and empirical investigations demonstrate an **insightful connection** between massive activations and the scaling behavior of AdaLN-zero (recognized by Reviewer ntGm).\n\n**2) A reasonable and practical training-free framework for feature extraction with DiTs.**\n\n- Motivated by the observed relationship between massive activations and the residual scaling factors of AdaLN-Zero, we propose a **reasonable and elegant** feature extraction framework, DiTF, which can be directly applied to existing pre-trained DiTs **without requiring any additional training** (recognized by Reviewer ntGm). Specifically, we reuse the built-in AdaLN modules in DiTs to effectively identify and regulate these massive activations with adaptive channel modulation, significantly enhancing the quality of representations.\n- Furthermore, **strategies for handling massive activations in LLMs [1] do not transfer effectively to DiTs**. Specifically, when directly applying the activation intervention strategy in LLMs [1] to DiTs, it fails to improve feature quality, as evidenced in Table i (29.9 vs. 30.1), highlighting the **fundamentally different nature** of massive activations in LLMs and DiTs. In contrast, our proposed DiTF enhances both feature semantics and discriminability, leading to substantial performance improvements on dense spatial tasks (**29.9 to 67.1**).\n\n| Model| Accuracy |\n| ----------------------------------- | :-------------------: |\n| Flux |29.9|\n| Flux + Activation intervention in LLMs [1] |30.1|\n| Flux + DiTF (Our)|67.1|\n\n*Table i. Performance comparison of Flux models with different massive activation tackling strategies on the SPair-71k semantic correspondence benchmark*\n\n**3) A novel perspective on the poor performance of DiT features in dense tasks.**\n- When directly using DiT features for dense spatial tasks, it yields disappointing results (Figure 1). In this paper, we identify the massive activations within DiT feature maps as **a key factor** limiting performance (recognized by Reviewer gPKV). Unlike Stable Diffusion (SD2-1), DiTs exhibit massive activations concentrated in a few fixed dimensions **across all image patch tokens** (Figure 2). These dominant dimensions cause **high directional similarity** among feature vectors [3], which makes it difficult to distinguish them using cosine similarity, and ultimately degrades performance on dense spatial tasks (Lines 35-39).\n\n**4) Novel insights into future DiT training.**\n- **REPresentation Alignment (REPA)**: REPA [4] aligns DiT hidden states with pretrained features (e.g., DINOv2) to facilitate training. However, due to the distinct feature characteristics: DiTs exhibit massive activations whereas DINOv2 does not, direct alignment may be suboptimal. Instead, aligning features after AdaLN modulation may  improve compatibility and alignment quality.\n\n\n### **Q2. On Theoretical depth and justification**\n\nWe respectfully clarify that our conclusions are not solely based on visual inspection or simple correlations, but are derived from a rigorous and systematic study across various pre-trained DiT models and are supported by both comprehensive results and theoretical insights. Specifically:\n\n**1) Robustness and generality.**\n\nWe examined a wide range of publicly available pre-trained DiT models, including SD3, SD3-5, Flux, etc., and consistently observed the same characteristics of massive activations. This consistent pattern suggests that our findings are not model-specific or incidental, but rather reflect a general property of DiTs.\n    \n**2) Comprehensive experimental and visual support.**\n\nOur paper conducts systematic experiments complemented by carefully designed visualizations to thoroughly investigate the phenomenon of massive activations. Furthermore, we provide plentiful supporting evidence in the supplementary materials, collectively establishing a robust experimental foundation for our conclusions.\n    \n**3) Theoretical justification for the proposed solution.**\n\nOur method is **theoretically motivated and interpretable**:\n- **Massive activation analysis**: We have provided theoretical justification for massive activation analysis in the introduction section (Lines 35-41). These massive activations share a few fixed dominant dimensions across all spatial tokens, resulting in a high degree of **directional similarity** for feature vectors. This property makes it challenging to distinguish spatial feature vectors using cosine similarity, which **hinders spatial discrimination of representation**. In addition, this analysis theoretically supports the conclusion that \"Massive activations hold little local information.\" \n- **Connection to AdaLN-Zero**: DiT feature updates are computed via residual connections: $z_{t}^{k+1} = z_{t}^{k} + \\alpha_k A_k(z_{t}^{k}),$ meaning AdaLN-zero directly affects the dominance and expressiveness of DiT features with residual scaling factor $\\alpha_k$. \n- **Channel-wise modulation with AdaLN**: Motivated by the above insights, we employ the built-in AdaLN to modulate massive activations in DiT features, formulated as: \n$$\n\\text{AdaLN}(z_{t}^{k}) = (1 + \\gamma_k) \\cdot \\text{LayerNorm}(z_{t}^{k}) + \\beta_k,\n$$ \nwhere $\\gamma_k \\in \\mathbb{R}^C$ and $\\beta_k \\in \\mathbb{R}^C$ are adaptive channel-wise scaling and shifting parameters. Specifically, AdaLN effectively **identifies and suppresses** dimensions associated with massive activations through the scaling factor $(1 + \\gamma_k)$, thus enhancing representation semantics and discrimination.\n\n### **Q3. Clarifying the Alleged Contradiction with 'Massive Activations in LLMs'**\n\nWe would like to clarify that our method **does not conflict with [1]**, as the **operational setting** and the **nature of massive activations** are fundamentally different.\n\n- In [1], the authors directly intervene in the hidden states during inference by manually setting massive activation values to zero in a single layer, and then **propagating the modified hidden state through the remaining model**. This intervention aims to probe the functional role of massive activations in the **internal computation** of LLMs. In contrast, our method treats the Diffusion Transformer (DiT) as a **frozen feature extractor**. We extract features from an intermediate block, apply AdaLN to these features, and perform a channel discard (zero out) operation to weakly massive activations in the **extracted features only**. Importantly, no modifications are made to the internal computation of the model.\n- Furthermore, the characteristics of massive activations in DiTs are **notably different** from those in LLMs [1]. The massive activations in LLMs [1] typically arise at **specific tokens** (e.g., starting token). In DiTs, such massive activations consistently occur at a **small number of fixed channel dimensions**, and appear **across all spatial tokens** (Figures 2 and 3). These **massive activations with concentration across all spatial tokens** in DiTs result in a high degree of **directional similarity** for feature vectors in the latent space [2]. Our proposed channel discarding strategy specifically addresses this issue by suppressing dominant but uninformative massive activations, thereby promoting feature discrimination and semantics.\n\n\n### **Q4. Generalizability on various CV tasks**\n\nWe have conducted **semantic segmentation** experiments on ADE20K (Table 6 of main text). Specifically, our approach boosts mIoU **from 43.8 to 53.6**. We further conduct **linear probing** on **ImageNet classification** using globally pooled DiT features, following the DINOv2 [3] protocol, and achieve a notable accuracy gain **from 65.8 to 72.6** (*DiTF$_{Flux}$*), highlighting the effectiveness and generalizability of our method.\n\n### **Q5. Typos and citations**\nThank you for your valuable feedback. In the final revision, we will correct all the typos and citation mistakes. In addition, we will revise our assumptions of AdaLN-Zero and AdaLN to accurately reflect their differences.\n\n[1] Massive activations in large language models, COLM 2024.\n\n[2] Clearclip: Decomposing clip representations for dense vision-language inference, ECCV 2024.\n\n[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2025.\n\n[4] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR2025."}, {"review_id": "oJyRLMx1DO", "reviewer": "Reviewer_gPKV", "summary": "This paper investigates Diffusion Transformers (DiT) and demonstrates that a slight test-time modification significantly boosts their performance on correspondence tasks. Firstly, the paper reveals the presence of massive activation in DiT features and shows that AdaLN alleviates this issue, leading to improved feature representation. Furthermore, the paper employs channel discarding to enhance these representations. The authors tested their method on various standard correspondence benchmarks and also show performance on two variants of DiT, namely SD3-5 and Flux.", "questions": "This is a very interesting paper with a strong presentation, and I support its acceptance. However, the reason why massive activation is harmful for representation remains unclear. I have suggested a hypothesis, which may or may not be correct, and would appreciate the authors' comments and a test of this hypothesis. Whether my hypothesis is right or wrong, if the authors can provide another persuasive hypothesis, I would be happy to raise my score.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\nI have always wondered why DiT features exhibit weak performance on correspondence tasks, and this paper provides a compelling explanation through intriguing and well-supported observations, presented in a clearly written text. The finding that massive activation in DiT harms its matching ability is particularly interesting.\n\n**Major Weaknesses**\n\nThe paper offers weak intuition regarding why features with massive activation are suboptimal for representation learning, as claimed in Line 166. The authors should at least elaborate on why this might be the case. Additionally, this argument could be considered an overstatement, given that diffusion features are typically well-learned from vast amounts of data. In my opinion, massive activation is specifically more detrimental to correspondence tasks. Typically, feature matching involves normalization followed by a dot product to construct cost volumes. My hypothesis is that since massive activation forms in the exact same channel for all features, the dot product between any features would be dominated by the square of this massive activation, leading to imprecise matching.\n\nBased on the hypothesis above, I wonder how the performance would be if channel discarding were applied before AdaLN. If my hypothesis is correct, then the crucial point is not whether pre-AdaLN or post-AdaLN is used, but rather that channel discarding is the most significant solution. If this shows similar performance to DiTF, then the methodology section should be slightly modified to de-emphasize the effect of AdaLN and instead highlight the effectiveness of channel discarding itself.\n\n**Minor Weaknesses**\n\n- In Line 73, the paper that used GAN should also be cited. \n- In Figure 5, the upper-left side of the activation visualization is partially cropped.\n- It would be great if the temporal correspondence results were highlighted in the main text, not just in the supplementary material.\n- Adding an explanation for why the geometric correspondence performance is lagging behind DIFT would be greatly appreciated. I suspect DIFT's high-resolution features might contribute to its strength in geometric correspondence.", "was_revised": true, "final_justification": "I greatly appreciate the authors' response. All of my concerns have been resolved. This is a good paper with excellent presentation and a highly interesting finding.", "rebuttal": "Thanks for your valuable suggestions. We’re glad that you found our work insightful and interesting. Please see the responses to your comments.\n\n### **Q1. Reason why massive activation is suboptimal for representation**\nThank you for your valuable comments. First, we would like to clarify that we have provided insightful intuition in the Introduction section (Lines 35–39), e.g., “These massive activations share a few fixed dominant dimensions, resulting in a high degree of **directional similarity** for feature vectors.” We are pleased to see that you agree with our explanation. To further investigate the effect of massive activations, we conduct detailed experiments on Table i, and we elaborate on several key findings based on the results.\n\n**1) Massive activations significantly hinder spatial discrimination** \n\nDue to the **directional homogeneity** introduced by massive activations, cosine similarity becomes ineffective in distinguishing spatial feature vectors. As shown in Table i, discarding the massive activation dimensions in feature maps significantly improves correspondence performance (from 29.9 to 37.5). Moreover, since DiTs are originally pretrained for generation tasks, their features often contain fine-grained visual details, which may be irrelevant or even detrimental for dense matching tasks [1]. For example, as illustrated in Figure 5 of the main text and Figures in the supplementary material, the original features (pre-AdaLN) tend to preserve vivid color cues as well as lighting and shading, which are **essential for generation but suboptimal** for visual correspondence (especially semantic correspondence). Consequently, the co-existence of massive activations and irrelevant details weakens the representational robustness of DiT features for correspondence tasks.\n\n**2) AdaLN enhances representation by suppressing activations and irrelevant details**  \n\nAs shown in Table i, directly applying channel discard to the original DiT features provides only suboptimal improvement, suggesting that reducing massive activations alone is insufficient. In contrast, AdaLN not only suppresses the massive activations but also effectively **activates other critical channels** essential for dense perception tasks. This dual effect mitigates the impact of massive activations while simultaneously **suppressing irrelevant low-level details** (e.g., color and shading), leading to a more refined and semantically meaningful feature space (see Figure 5 and the supplementary material). Consequently, the collaboration between AdaLN and channel discard operation leads to substantial gains in performance (**29.9 to 67.1**).\n\n| Model| Accuracy |\n| ----------------------------------- | :-------------------: |\n| Original|29.9|\n| with Channel discard |37.5|\n| with AdaLN|65.3|\n| with AdaLN & Channel discard|67.1|\n\n*Table i. Ablation study of our model DiTF$_{Flux}$ on the SPair-71k semantic correspondence benchmark.*\n\n\n### **Q2. Explanation for the geometric correspondence performance**\n\nThank you for your comments. We agree that feature resolution plays a critical role in determining geometric correspondence performance. Specifically, given an input image of size $H \\times W$, our DiTF framework produces features at a resolution of $\\frac{H}{16} \\times \\frac{W}{16}$. In contrast, DIFT [2] leverages higher-resolution features of size $\\frac{H}{8} \\times \\frac{W}{8}$, which are extracted from its 2nd upsampling block. These higher-resolution features preserve finer-grained texture and structural details, thereby contributing to better geometric localization. As shown in Table ii, when DIFT utilizes the lower-resolution features of size $\\frac{H}{16} \\times \\frac{W}{16}$ from its 1st upsampling block, it achieves a reduced accuracy of 37.2%, which is lower than our DiTF$_{Flux}$ (41.9%). This comparison underscores the effectiveness of our framework.\n\n| Model | Resolution| Accuracy |\n| :-------------: | :-------------------: | :-------------------: |\n| DIFT |$\\frac{H}{8} \\times \\frac{W}{8}$|45.6|\n| DIFT|$\\frac{H}{16} \\times \\frac{W}{16}$|37.2|\n| DiTF$_{Flux}$ (Our)|$\\frac{H}{16} \\times \\frac{W}{16}$|41.9|\n\n*Table ii. Comparisons between DIFT and Our DiTF$_{Flux}$ model on the HPatches geometric correspondence benchmark.*\n\n### **Q3. Typos and citations**\n\nThank you for your feedback. In the final version, we will properly cite the related GAN literature and fix all typos and errors in Figure 5. In addition, we will highlight temporal correspondence results in the main text.\n\n[1] Representation alignment for generation: Training diffusion transformers is easier than you think, ICLR 2025.\n\n[2] Emergent Correspondence from Image Diffusion, NeurIPS 2023."}]

Author rebuttal:

Meta-review: This paper investigates Diffusion Transformers (DiTs) for dense visual correspondence and highlights the phenomenon of massive activations, where a small number of feature activations exhibit disproportionately large values. Such activations degrade feature quality and hinder performance. To address this issue, the authors introduce Diffusion Transformer Feature (DiTF), a training-free framework that employs AdaLN modulation and channel discard to suppress these activations.

The strengths of the work lie in its systematic analysis of massive activations, its practical and training-free solution, and its comprehensive experimental validation across diverse architectures and datasets. Reviewers emphasized that the framework is elegant, readily applicable to existing pretrained DiTs, and delivers significant performance improvements.

After the rebuttal, most concerns were satisfactorily resolved. The authors provided theoretical insights into activation differences among LLMs, ViTs, and DiTs, clarified the computational efficiency of their method, and demonstrated robustness across hyperparameter settings. They also acknowledged related work (TinyFusion) and committed to revising their novelty claims, while presenting additional experiments on segmentation and classification to support generalization further.

In conclusion, this paper offers a solid contribution backed by substantial empirical evidence. I recommend acceptance and encourage the authors to incorporate the reviewers’ suggested revisions into the final version.
Decision: Accept (poster)