Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers
Authors: Chaofan Gan
Zicheng Zhao
Yuanpeng Tu
Xi Chen
Ziran Qin
Tieyuan Chen
Mehrtash Harandi
Weiyao Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Massive Activations
Diffusion Transformers
Visual Detail Synthesis
Primary area: generative models

Abstract: Massive Activations (MAs) are a well-documented phenomenon across Transformer architectures, and prior studies in both LLMs and ViTs have shown that they play a substantial role in shaping model behavior. However, the nature and function of MAs within Diffusion Transformers (DiTs) remain largely unexplored. In this work, we systematically investigate these activations to elucidate their role in visual generation. We found that these massive activations occur across all spatial tokens, and their distribution is modulated by the input timestep embeddings. Importantly, our investigations further demonstrate that these massive activations play a key role in local detail synthesis, while having minimal impact on the overall semantic content of output.
Building on these insights, we propose Detail Guidance (DG), a MAs-driven, training-free self-guidance strategy to explicitly enhance local detail fidelity for DiTs. Specifically, DG constructs a degraded ``detail-deficient'' model by disrupting MAs and leverages it to guide the original network toward higher-quality detail synthesis. Our DG can seamlessly integrate with Classifier-Free Guidance (CFG), enabling joint enhancement of detail fidelity and prompt alignment. Extensive experiments demonstrate that our DG consistently improves local detail quality across various pre-trained DiTs (\eg, SD3, SD3.5, and Flux).

Published as a conference paper at ICLR 2026
MASSIVE ACTIVATIONS ARE THE KEY TO LOCAL DE- TAIL S SYNTHESIS IN DIFFUSION TRANSFORMERS
Chaofan Gan 1.2
Zicheng Zhao'
Yuanpeng Tu
Xi Chen
Ziran Qin'
Tieyuan Chen'
Mehrtash Harandi
Weiyao Lin
Shanghai Jiao Tong University, Monash University, The University of Hong Kong
Detall Guidance
Detail Guidance (DG) Enhances CFC Details
CPC
CPC + DG
CFC
CFO + DO
Figure 1: Visual results of our Detail Guidance (DG). Left: DG explicitly enhances fine-grained visual details, yielding high-quality outputs. Right: DG integrates seamlessly with Classifier-Free Guidance (CFG), allowing for further refinement of details.
ABSTRACT
Massive Activations (MAs) are well-documented phenomenon across Trans- former architectures, and prior studies in both LLMs and ViTs have shown that they play substantial role shaping model behavior. However, the nature and function of MAs within Diffusion Transformers (DiTs) remain largely unexplored. In this work, we systematically investigate these activations to elucidate their role in visual generation. We found that these massive activations tial tokens, and their distribution is modulated by the input timestep embeddings. Importantly, our investigations further demonstrate that these massive activations play a key role in local detail synthesis, while having minimal impact on the overall semantic content of output. Building these insights, we propose Detail Guidance (DG). a MAs- driven,, training-fre free self-guidance strategy to explicitly enhance local detail fidelity for DiTs. Specifically. DG constructs degraded model by disrupting MAs and leverages it to guide the original network toward higher-quality detail synthesis. Our DG can seamlessly integrate with Classifier- Free Guidance (CFG), enabling joint enhancement of detail fidelity and prompt alignment. Extensive experiments demonstrate that our DG consistently improves local detail quality across various pre-trained DiTs (e.g., SD3, SD3.5, and Flux).
across all spa-
INTRODUCTION
Diffusion models (Rombach et al. 2022 Saharia et al. 2022) have recently achieved remarkable suc- cess across a wide range of generative tasks. Among various architectures, the Transformer (Vaswani
"Corresponding Author.
Project page: httpttts////tt /ttt:ss://ganccaofaooo000 gi thub. 1o/DG

---

Published as a conference paper at ICLR 2026
200
20
2000
150
200
2000
1000
Patch takans
s0 1o0 200250
292
Dim
Patch tokens
2000
2000
acoc
810
Dkm
Patch takens
2009
2000
4000
470
Dmm
1000 good
Patch takens
2002 400.
1440
Dim
DIT-XL/2
SD3
SD3.5
F Flux
Figure 2: Massive Activations in DiTs. The activation magnitudes of internal hidden states from the middle block (A: = N/2) and timestep (t = T/2). We present the average magnitudes over 1,000 text prompts. Massive Activations (MAs) are consistently concentrated in a few fixed dimensions across all image patch tokens. The MA dimensions remain consistent across all layers (see Figure 14)
et al. 2017) has emerged as powerful and versatile backbone for diffusion models (Peebles & Xie p.23p, thanks to flexibility and scalability. With the increasing availability of large-scale data and computational resources, many large Diffusion Transformers (DiTs) (Peebles & Xiel [0233 Esser
cct al. 2024) have recently emerged, achieving state-of-the-an -rtt performance in both image and video synthesis YYngg et al. 2024b Hong al. 2022
Wan et al. 2025).
Along with the rapid progress of DiTs, recent studies (Sun et al. 2024 Darcet et al. 2024 Gan
et al. 2025) have uncovered interesting phenomenon known Massive Activations (MAs) in these Transformer- based models, where rare hidden activations exhibit unusually large magnitudes.
Specifically, (Sun al., 2024 Xiao et al. 2024) identifies the massive activations in Large Language Models (LL.M) and demonstrates that they are essential for long-context learning. Similar activation patterns are observed in Vision Transformers (ViTs), where they are utilized process global semantic information (Darcet et al. 2024). More recently, several works (Gan et al. 2025 Fang jet al. 2025) have reported the presence of massive activations DìT's However, their functional role within the visual generation process of DiTs remains largely unexplored.
In this paper, we aim to gain a deeper understanding of the role massive activations play in the visual generation tasks. We first conduct systematic investigations to study the characteristics of massive activations. Our investigations reveal that massive activations appear in few fixed dimensions across all image tokens, which are text-independent ( Figures and 4 In addition, we demonstrate that these activations are closely associated with the input timestep embeddings, where the timestep encoding can directly shape its distribution (Figure 4
Furthermore, we perform activation intervention by disrupting the internal massive activations to directly investigate their impact DiT generation. Our analysis shows that, when disrupting the massive activations, the visual output preserves consistent semantic content with the original images (Figure These results suggest that the massive activations exert minimal influence on the semantics of the generation process. On the other hand, we found that the local details of visual output are significantly degraded when massive activations are disrupted, suggesting their crucial role in local detail synthesis (Figure 5). We propose the following interpretation to these findings: DiT assigns massive activations all spatial tokens to drive fine-grainet local detail synthesis of each token, while timestep embeddings modulate these activations to adaptively control the detail synthesis process throughout generation.
Motivated by these insights, we introduce Detail Guidance (DG), a MAs-driven, training-free self- guidance strategy for detail enhancement in DiT' generation. Specifically, we construct a degraded ddettal-deficieme network by disrupting the massive activations, and then leverage it to explicitly guide the original model toward generating higher-quality details. Our approach can be seamlessly integrated with classifier-free guidance (CFG), thereby achieving joint enhancement of detail fidelity and prompt alignment. Our main contributions can be summarized as follows.
We provide a comprehensive study of massive activations in DiTs, demonstrating that these activations are crucial for fine-grained local detail synthesis while exerting minimal influence on the overall semantic content.
We trace the massive activations to the influence of the input timestep embeddings, revealing that the input timestep encoding can directly shape their distribution.
2

---

Published as a conference paper at ICLR 2026
We introduce Detail Guidance (DG), MA-driven, training- ffre self self-guidance strategy to explicitly enhance local detail synthesis in DiTs. DG integrates seamlessly with Classifier Free Guidance (CFG), leading to improved local detail synthesis.
2 RELATED WORK
2.1 DIFFUSION MODEL
Diffusion models (Rombach et al. 2022 Dhariwal & Nichol, 2021 Saharia et al. 2022) have
become a dominant paradigm for high-quality visual synthesis. Harly approaches primarily relied on U-Net architectures (Rombach et al. 2022) to model the denoising process. Recently, the field has shifted toward Transformer- bassrdeerrrareddddd backbones (Vaswani et al., 2017). Among these advances, Diffusion Transformers (DiTs) (Peebles & Xie 2023) have rapidly established themselves powerful backbone for visual generation. Due to the strong scalability and flexibility of transformer architecture,
wave of large-scale DiTs(Esser et al., 2024 black-forest labs] (0024) (e.g., SD3,
Flux) has emerged, achieving superior performance in various visual generation tasks YYnn et al.
2024b
Wan et al. 2025).
2.2
MASSIVE ACTIVATIONS
Xu al. 2023) have identified the presence of massive activations large language models (LLMs).. These activations typically emerge fixed dimensions of con-information tokens, such as starting or delimiter tokens. Importantly, some works (Xiao et al., 2024 Jin et al. 2025) have shown that massive activations contribute positively to contextual knowledge modeling. enabling LLMs to capture long-range dependencies more effectively. In addition, (Jin et al. (2255) traced the emergence of concentrated massive values into rotary position embeddings (RoPE).
Massive activations in LLMs. Recent studies (Sun et al., 2024 Zhao al. 2023
Massive activations in ViTs. Similar activation patterns have also been observed in Vision Trans- formers (ViTs) (Darcet al. 2024 Yang al. 2024a (Sun et al. 2024). In ViT', massive activations frequently arise in redundant background tokens and have been associated with encoding global semantic information. Moreover, (Yang et al. 2024a) traced the emergence these activations to the influence of the input positional embeddings.
Massive activations in DiTs. Several studies on the acceleration of Diffusion Transformers (DiTs) (Liu & Zhang, 2024 Fang et al., 2025 Zhao al. 20244 have highlighted the presence of outlier activations, whose extreme values pose a significant challenge for model quantization and distillation. More recently, DiTF (Gan et al. 2025) found that massive activations occur at fixed dimensions across all spatial tokens when employing DiT's as feature extractors, and showed that these activations substantially influence the discriminative quality of extracted features. However, the function of these massive activations in visual generation remains largely unexplored.
2.3 SAMPLING GUIDANCE FOR DIFFUSION MODELS.
Classifier-free guidance (CFG) (Ho & Salimans, 2022 has become the standard guidance mechanism for diffusion sampling. It extrapolates between the conditional and unconditional branches amplify conditioning signals, thereby enhancing controllability and improving semantic alignment. Recently, number of advanced strategies (Fan al. 2025 Chung et al.) (2024; Shen et al.) 2024: Sadat et al.) 2024; /Zhang et al. 2024) have been proposed to further improve its effectiveness. For example, adaativve guidance strengths for different semantic components, while (Zhang et al. 2024) propose frequency-awan guidance mechanism that adaptively modulates different frequency bands. Beyond CFG, auto-guidance (Karras et al.] 2024) introduced a self- guidance signal by guiding the base model with deliberately degraded "bad" version (e.g., reduced capacity or under-traine
(Shen al. 2024) introduces
checkpoints), steering sampling toward higher-qualis outputs. Other approaches (Ahn et al., 2024 Hong al.) 2023 Hyung et al.) (2025) construct degraded predictions by perturbing internal mechanisms such as modifying attention maps or skipping blocks to bias the sampler toward better image distribution.
3

---

Published as a conference paper at ICLR 2026
Activaaionn value
500
10
Activatoon value
500-
10
Activation value
500-
10
50
proportion
50
10
proport
50-
10a
proportion
10-a
10
10
0.1-
10
0.1
22
S0k
100k
250k
400k
iterations
arch
(a) Activations along layers
(b) Activations along iterations
(c) Activations across model size
Figure Illustration of several properties of massive activations DiT-XL. (a) Activation distribution of the hidden states along DiT layers (b) Activation distribution of the hidden states along training iterations (c) Activation distribution of the hidden states across different model sizes. Massive activations occur throughout all layers and persist across different model sizes.
PRELIMINARIES
Diffusion models. Diffusion models (Ho et al. 2020; Karras et al. 2022) generate data by progres- sively denoising Gaussian noise, starting from žт ~ (0, 1). Given clean sample x ~ Pama (a)), the forward diffusion process be expressed 24 + (t)) e, where att) denotes the noise schedule and ~ N(0, I). To learn the reverse process, a denoising network D2(2,, f, c) is trained to predict the injected noise each step, where is the conditioning signals (e.g., class labels or text prompts).
Classifier-Free Guidance. Classifier-Free Guidance (CFG) (Ho & Salimans, 2022) enhances diffusion model quality by jointly training the denoising network in conditional Do(2,, c) and unconditional (2(2,, t) modes. At sampling time, combines the two predictions to amplify the conditioning signal:
(2,,t Da (2,, + (Do (2,,t,c) - De (2t, t))
(I)
where w is the guidance scale. By extrapolating their difference, CFG strengthens semantic alignment and improves generation fidelity, but can sometimes lead to insufficient synthesis of fine- enaignai local details (Sadat et al.) 2024 Chung et al. 2024
DiT architecture. We provide the architecture of Diffusion Transformer (DiT) following (Peebles Xie 2023). For clarity, we omit the VAE component and focus on the latent diffusion transformer, denoted as Dg = [D] where & indexes the block and N the total number of blocks. Given noised latent 2n Cx the DiT block D3 forward the internal hidden state of through a residual connection (He et al.) 2016) to the next block, formulated
* 2, + DxD t,cc
(2)
where ag E RCC denotes the dimension-wise residual scaling factor derived from the AdaLN layer (Perez et al. 2018). More architecture details can be found in Appendix
4 MASSIVE ACTIVATIONS IN DIFFUSION TRANSFORMERS
As shown in Figure 2 the hidden states of various DìTs consistently exhibit prominent phenomenon: Massive Activations (MAs). This observation suggests that massive activations must play a crucial role the visual generation process of DiTs. this section, we conduct an in-depth investigation to understand why and where these massive activations emerge, and analyze their role in the visual generation process of DiTs.
4.1 CHARACTERISTICS OF MASSIVE ACTIVATIONS
Massive activations occur throughout all layers across different model sizes. We first investigate where massive activations emerge. As shown in Figure we observed that massive activations exist in all internal DiT layers(Figure a)). They emerge early during training (before 50k training iterations in Figure b))),, underscoring their crucial role in the internal computations of DiTs. Moreover, massive activations persist across models of different scales (Figure c)). We present the layer properties SD3, SD3.5, and Flux in Appendix B which further confirm their presence throughout all internal blocks.
4

---

Published as a conference paper at ICLR 2026
300
10
170
Activtion value
50
Activation Scaling fedor
Activation value
160
20
Activattonn
0.2
1/0
3154
172
74
230
535
1000
dimension
timestep
text
(a) Scaling factor impact activations
(b) Timestep text impact on MAs
Figure 4: Impact of the input timestep and text Massive Activations (MAs) in SD3. (a) Comparison of the distributions of hidden-state z activations and their corresponding residual scaling factor Ork- (b) Respective impact of input timestep and text embeddings the magnitude distribution of MAs, where we compare the MAs of 1000 different text inputs. The massive activations are governed by the residual scaling factor; their magnitude primarily shaped by the input timestep embedding f, while text embeddings e have negligible effect.
Massive activations appear in fixed dimensions across all patch tokens. Then, we amalyze the spatial distribution of massive activations, illustrated in Figure 2 The results reveal that massive activations consistently appear at fixed feature dimension (e.g., dimension 810 for SD3) across all spatial tokens. DiTF (Gan et 2025) has also characterized similar properties.
To delve into the massive activations in hidden states, we first examine the computation of hidden states = IR Cx
k+1 -+ = on De (,, MT.P
where hidden states are computed via a residual connection, and or E gC denotes the dimension- wise scaling factor parameters regressed by the AdaLN layer with MLP network (see Appendix A for details). As shown in Figure 4[a, ()., we compare the activation distributions hidden states 2f and the scaling factor 0g across dimensions. It can be observed that prominent peak of ag dimension 810 leads to a corresponding concentration of massive activations (Figure aa)), indicating that the scaling factor or governs the dimension pattern of massive activations.
As the scaling factor og is produced by the Adal.N layer conditioned on the input timestep embedding and text embedding e (see Equation 3), we further examine how and e respectively influence MAs (Figure 4b). This analysis leads to two key observations:
Text embeddings have minimal impact massive activations. As shown in Figure bb), compare the massive activation value across 1,000 different text prompts. We observe that these activations remain nearly constant (around 150) regardless of the input text embeddings. indicating that the text embeddings have negligible influence on the magnitude of the massive activations.
Timestep embeddings shape the massive activations. In contrast, find that the timestep plays a dominant role for massive activation: the magnitude of massive activations increases steadily as decreases from T O. We also get similar observations for SD3-5 and Flux (see Appendix C These results suggest that massive activations in DiTs are mainly modulated by the timestep embeddings.
4.2 MASSIVE ACTIVATIONS FOR LOCAL DETAIL. SYNTHESIS
Previous works (Darcet et al. 2024 have characterized massive activations in ViTs, showing that they primarily arise in specific tokens (c.g.. background tokens) and serve to encode global information. In contrast, massive activations DiT's occur across all spatial tokens. This fundamental difference naturally raises key research question: What role do massive activations play in DiTs? To address this question, we perform activation intervention (Sun et al. 2024) to examine how massive activa- tions influence the behavior of DiTs. Specifically,
manually disrupt the massive activation values single layer and then propagate the modified hidden state through the remaining DiT blocks. The results presented in Figure 5 We provide the full activation intervention setting, including original. Non-MAs disrupted, and MAs-disrupted Appendix D
Massive activations have minimal impact on semantic content. We observe that the images generated by the MAs-disrupted model still preserve global semantics such object identity, color composition, and overall layout, remaining consistent with those from the original model (Figure Moreover, the MAs-disrupted model maintains comparable prompt alignment metrics, achieving
5

---

Published as a conference paper at ICLR 2026
Original
Disrupted
0:538
0.982
Cipckore
0 1:486
0532
5D3
HP5K2
0.024
BipSetre
0:354
U2 Cipccer
0.514
6/185
5:967
Origial
Disrupted
(a) Visual comparison
(b) Win probability comparison
Figure 5: Comparison of the original and Massive Activations (MAs) disrupted models. (a) Sampling results comparison between the original and MAs-disrupted models for SD3. (b) Win probability comparison for different models where we evaluate the model from two perspectives: Prompt Alignment (Blipscore and Clipscore) and Local Detail Quality (HPSv2.1 I and Laion- Aesthetics). Disrupting massive activations markedly degrades the fidelity of fine-grained details in the generated images while exerting minimal impact semantic content.
similar Blipscore [L al. 2022) (0.462 vs. 0.538) and Clipscore (Radford et al. 2021 (0.512 vs. 0,488) win probabilities relative to the original output (Figure (b)). These results indicate that the inherent massive activations exert minimal influence on the overall semantic content the visual generation process of DiT's This finding is consistent with the characteristic described in Section where the input text embedding has negligible effect on massive activations.
Massive activations play a key role local detail synthesis. More importantly, it can be easily observed that the fine-grained local details, including textures and subtle object parts (e.g.. eyes and hair), are markedly degraded when massive activations are disrupted. Moreover, the MAs-disrupted model attains much lower win probabilities (0.028 on HPSv2.1 and 0.078 Laion-Aesthetics) than the original model on the local detail quality metric, underscoring the crucial role of massive activations in fine-grained detail synthesis.
In combination with the characteristics of MAS described in Section 4.1 we summarize two key findings: (1) MAs are mainly shaped by the input timestep embedding, and (2) they crucial for local detail synthesis. These findings are consistent with the generative dynamics of diffusion models (Ho et al., 2020) Rombach al. 2022): the timestep embedding encodes the noise level and generation stage, with large guiding coarse structure reconstruction and small t driving fine- grained refinement. As sampling proceeds from to modulates the residual scaling factor Oik progressively amplifying massive activations (Figure 4b)), which in turn orchestrate the detail synthesis process in DiTs.
4.3 DETAIL GUIDANCE FOR DIFFUSION TRANSFORMERS
Based on these findings, we make the following hypothesis: during training, DiT learns to assign massive activations to all spatial tokens to drive fine-grained local detail synthesis of each token, and uses timestep embeddings to modulate massive activations, thereby adaptively orchestrating the detail synthesis process throughout generation.
Motivated by these insights, we seek concise and effective approach to exploit the capacity of MAs for enhancing fine-grained detail synthesis in DiTs. Accordingly, we propose MAs-driven, training- free self- self-guidance strategy, termed Detail Guidance (DG). Our approach draws inspiration from the self-guidande mechanism (Karras et al.) 2024), which guides the base model with deliberately degraded "bad" version. Different from them, we construct the "bad" model by explicitly degrading its capacity for local detail synthesis.
Formally, let Dg be the original pretrained DiT model and of E CCCH the hidden state output of k-th block. We disrupt the massive activations in 2 by masking (zeroing out) the corresponding dimensions to the massive activations and then propagate the modified hidden state z/ through the remaining blocks. By disrupting the massive activations (drivers of local detail), build degraded

---

Published as a conference paper at ICLR 2026
Baseline
Baseline
house in the style of Eacker
hage wide open plao house many couches arranged dircee
tulps leaver Rowerss all pointing seomin "ase"" "senberpice"
cuble borling on cboud of colorfai brees
Har weary hamtsbr soldier
somtuaal pizza cod
onime girl cyes standing pite Fireso Hearitg
Figure 6: Visual results Detail Guidance (DG) on SD3. Baseline denotes visual outputs without CFG. DG produces high-quality images with improved fine-graid fine-grained details compared to Baseline. The CFG output is included as a reference for better comparison detail quality.
model Da.m that produces detail-deficienl outputs, where m is the disrupted layer depth. Leveraging this degraded "detail-deficien" model De.ms m. we formulate our Detail Guidance (DG) following the diffusion self-guidance mechanism (Krrras et al. 2024):
Do(2,, t, c) = Do (2,, t, c) + w (Doozz,t,e) - Do.m (2e, t, c))
where controls the strength of the detail guidance. Our approach requires extra training and can be directly applied to mostly pretrained DiT models (Table D..
Integration with CFG. Classifier- free guidance (CFG) (Ho & Salimans 2022 is a standard
technique that enhances semantic alignment by extrapolating between conditional and unconditional predictions. Our DG method naturally complementary to CFG: whereas CFG strengthens semantic fidelity, DG explicitly enhances local detail quality. The combined guidance is expressed as
1, Do(z,, c) A(Dn(ze; (, Do(2,, !))
(5)
+ w (Do(zz, t, c) Do t,,,
where À and are the guidance scales of CFG and DG, respectively.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Model Variants. As our approach merely modifies internal hidden states, can be readily applied to most pretrained DiTs without additional training or tuning. We evaluate DG three representative text-to-image DiTs, SD3-Medium (Esser et al. 2024 (SD3). SD3.5- (Esser et al.) 2024) (SD3.5), and Flux (black-forest labs 2024). To comprehensively assess its effectiveness, test DG under two settings: Conditional (Cond) generation without CFG and CFG generation. The default generated image size is 1024x 1024. Further implementation details provided in Appendix H.2
7

---

Published as a conference paper at ICLR 2026
CFG
CFC + DG
CPG
CFC DG
Figure Integration with CFG. Rows 1-2: SD3; Rows 3-4: SD3.5. Incorporating DG into
Classifier-Fres Guidance (CFG) yields outputs with markedly richer and more refined local details.
Prompt Aligmment
Detail Quality
Model
Type
DG
Clipscore
Blipscore
HPSv2.
Aesthetic
x
22.
66.74
21.84
5.58
Cond
V
24.15
76.52
28.65
6.01
SD3
x
26.64
87.01
28.23
5.80
CFG
V
26.25
86.32
29.87
6.03
X
24.90
70.09
23.65
5.94
Cond
V
26.01
83.66
29.23
6.16
SD3.5
X
27.67
92.62
29.9
6.01
CFG
27.68
91.61
30.7
6.18
22.09
57.60
19.33
5.50
Cond
V
25.69
80.55
27.88
6.13
Flux
X
27.04
87.76
29.16
5.96
CFG
27.14
86.23
29.25
6.12
Table 1: Quantitative comparison on dataset Pick- a-Pic. Our DG strategy brings substantial improvements on detail quality for both settings, demonstrating its effectiveness in enhancing visual details. The best highlights in bold.
Datasets and Evaluation Metric. We assess our method on two standard benchmarks: the Pick-a-Pic
"test unique" split (Kirstain al. 2023) and HPSv2. (Wu et al. 2023). To quantify prompt alignment,
we compute Clipscore (Radford et al. 202T) and Blipscore (Li et al. 2022). To evaluate the fidelity
of fine- -oe---raind local details, we adopt HPSv2.1 (Wu et al. 2023) and -aith-Asstestiss (Schuhmann
2022) as quality metrics. Further evaluation details are provided in Appendix H.3
5.2
MAIN RESULTS ON VISUAL GENERATION
Evaluation of DG. Table D reports the quantitative results of Detail Guidance (DG) three pre- trained DiTs. DG achieves substantial improvements in both prompt alignment and detail quality, (e.g., Blipscore from 70.09 83.66 and Aesthetic from 5.94 to 6.16 SD3.5). Qualitative results in Figure 66 further confirm that DG effectively enhances ane-ggrined local details while faithfully pre- serving the overall image structure. We provide qualitative results SD3.5 and Flux in Appendix P
8

---

Published as a conference paper at ICLR 2026
HPSv2.
Train
Method
Aesthetic
Anime
Concept
Painting
Photo
Avg.
CFG
31.34
30.62
30.98
28.01
30.24
5.93
APG
30.76
29.98
30.24
26.86
29.46
5.89
CFG++
31.58
30.32
30.95
27.24
30.02
5.91
Semantic-CFG
30.92
29.99
30.92
29.16
30.25
5.89
FA-CFO
31.07
30.10
31.09
28.76
30.26
5.96
CFG-Zero
31.64
31.05
31.35
28.25
30.57
6.07
PAG
30.59
28.92
29.38
27.91
29.20
6.10
X
DG (Ours)
31.14
30.17
30.05
28.70
30.14
6.14
CFG+DG (Ours)
32.23
31.11
31.27
29.21
30.96
6.13
Table 2:
Evaluation of different guidance on dataset HPSv2.1 with SD3. Train means whether need to train an unconditional branch. The best highlights in bold, while the second best is underlined.
CFG
DG
CFG DG
Aesthetic
(a)
Aesthetic
(b)
Clipscore
(c)
Aesthetic
(d)
27
3.7
5.7
23
5.7
5.5
10 12
16
16
5.5
0.0
0.5
1.0
2.0
25
300
21 0.0 0.5 10
2. 2.5
5.5
2
Disrupted depth (m)
Guidance scale (1))
Guidunce scale (w)
Guidance scule (A)
Figure 8: Investigations of disrupted depth m, scales and w for SD3.
Moreover, we report experiments on ImageNet 256x256 in Appendix showing that our DG strategy also enhances visual quality consiionniiional generation tasks.
DG versus CFG. From Table DG yields higher detail-quality scores (e.g., Aesthetic 6.01 vs. 5.80 for SD3), whereas CFG achieves stronger prompt alignment. As illustrated Figure DG produces outputs with richer local textures, while CFG excels at semantic alignment. These results indicate that DG primarily enhances local detail synthesis, while CFG strengthens semantic alignment.
Integrating DG with CFG. DG integrates seamlessly with CFG, consistently improving detail quality as shown Table [ Visual comparisons Figure 7 highlight that combined strategy further refines fine-graing grained details, yielding higher overall image quality. We provide more visual results of SD3 and SD3.5 in Appendix(O
DG versus PAG. PAG generates the bad version by modifying the self-attention maps of the conditional branch and does not require training an unconditional branch. As shown in Table 2 DG achieves better performance than PAG (e.g., 30.14 14 vs. 29.20 on HPSv2. and 6.14 vs. 6.10 Aesthetic), demonstrating its stronger effectiveness in enhancing visual details.
DG versus advanced CFG. We further compare DG with advanced CFG strategies such as FA-CFG, Semantic-CFG, and CFG-Zero (see Table 2 DG requires no inconditional--rant training, yet still obtains the highest Aesthetic score (6. 14) and competitive 1PSv2. performance among all methods. When combined with CFG, DG achieves the best results on both HPSv2. and Aesthetic metrics, underscoring effectiveness and generality improving visual quality.
5.3 ABLATION STUDY
Disrupted depth m. We examine the effect of the disrupted depth of massive activations, as shown in Figure 3 a). Our DG strategy achieves the best performance when applied to intermediate blocks (e.g., ranging from 4 to 10). We hypothesize that early blocks mainly contain heavy noise and lack even coarse image structures, making disruption there uninformative, while applying disruption in late blocks occurs too close the final output and thus has minor impact generation. Based these observations, we primarily perturb massive activations in the intermediate layers and set the default for the SD3 model. The configurations for SD3.5 and Flux provided in Appendix H.2
Guidance scales w and We present the quantitative results across different scales in Figure Our DG consistently achieves stable and high Aesthetic (AES) scores (Figure (()) and CLIPScore

---

Published as a conference paper at ICLR 2026
Baseline
Local DG
DG
Figure Visual results of our Local DG on SD3. We implement the Local DG strategy by selectively disrupting the MAs in tokens within a target region (e.g.. the dog). Local DG significantly enhances the details of the local region (e.g.. dog in the first row) while leaving other regions, such as the cat and background, essentially unaffected.
(Figure (c)). When combined with CFG, it further boosts AES performance (Figure ())). These results highlight the effectiveness and robustness our approach in enhancing fine-grined details.
User study. We conduct user study to evaluate the benefits of our DG strategy from three key aspects: prompt alignment, color consistency, and detail preservation(see Appendix ( The results show that DG strategy yields substantial improvements in color consistency and detail preservation, demonstrating its effectiveness enhancing fine- grained visual details.
5.4 TOKEN-SPECIFIC GUIDANCE USING LOCAL DG
To further investigate the locality property of MAs, we design a Local DG strategy in which we mask MAs exclusively for selected subset of spatial tokens, rather than masking MAs across all spatial tokens as in the original DG setting. As shown in Figure 9 Local DG effectively guides local details in the generated images. Specifically, by disrupting the Massive Activations only in the tokens corresponding to the dog, Local DG successfully enhances the local details of the dog while leaving other regions, such the and background, essentially unaffected. These results clearly demonstrate the locality of massive activations: each MA primarily drives the local detail synthesis of its corresponding spatial token. By exploiting this property, we can achieve token-specific guidance using DG by selectively disrupting MAs in the target regions.
CONCLUSION
In this paper, we systematically investigate an intriguing phenomenon DiTs. termed Massive Activations (MAs). We find that these activations emerge across all spatial tokens and that their distribution is shaped by the input timestep embeddings. Our further analysis demonstrates that these activations are critical for local detail synthesis in DiT's. We interpret them as drivers of local detail information whose magnitude is dynamically modulated by timestep embeddings, thereby orchestrat- ing detail synthesis during the DiT generation process. Building on these insights, we propose Detail Guidance (DG), MAs-driven, training-free free self guidanee strategy to explicitly enhance local detail synthesis. Our DG can be seamlessly combined with CFG, enabling joint enhancement of detail fidelity and prompt alignment. Extensive experiments demonstrate the effectiveness our approach in improving fine-grained detail synthesis.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
The paper is supported in part by the National Natural Science Foundation of China (No. 62325109, 62561 160155, 62595733), and in part by the Shanghai 'The Belt and Road" Young Scholar Exchange Grant (24510742000). Mehrtash Harandi is supported by the Australian Research Council (ARC) Discovery Program DP250100262.
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang. Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. European Conference on Computer Vision, pp. 1-17. Springer, 2024.
black-forest
labs.
Flux.
https::/blackfooest..aaa
ancouncing-black Sortacck-orret--eee 2024.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. urXiv:2310.00422 2023.
Hyungjin Chung. Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. Cfg++: Manifold- constrained classifier free guidance for diffusion models. arXiv preprint (XXi:22406008000, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference Learning Representations, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024.
Weichen Fan, Amber Yijia Zheng, Raymond A Yeh, and Ziwei Liu. Cfg-zero*: Improved classifier- free guidance for flow matching models. arXiv preprint arXiv:2503. 18886, 2025.
Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
18144-18155 2025.
Chaofan Gan, Yuanpeng Tu, Xi Chen, Tieyuan Chen, Yuxi Li, Mehrtash Harandi, and Weiyao Lin. Unleashing diffusion transformers for visual correspondence by modulating massive activations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXhv:2207.1259% 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Susung Hong, Gyuseong Lee, Wooseok Jang, and Scungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7462-7471, 2023.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 2022.
Junha Hyung, Kinam Kim, Susung Hong, Min-Jung Kim, and Jacgul Choo. Spatiotemporal skip guidance for enhanced video diffusion sampling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 11006-11015, 2025.

---

Published as a conference paper at ICLR 2026
Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules are the key to contextual knowledge understanding. arXiv preprint arXiv:2502.01550 2025.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Tero Karras, Miik Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with a bad version of itself. Advances in Neural Information Processing Systems, 37:52996-53021, 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances neural information processing systems, (6633652--366633 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified Asion-language understanding and generation. In International conference on machine learning, pp. 12888-12909 PMLR, 2022.
Wenxuan Liu and Sai Qian Zhang. Hq-dit: Efficient diffusion transformer with fp4 hybrid quantization. arXiv preprint arXiv:2405. /9754455.177,, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
E Perez, F Strub, H De Vries, V Dumoulin, and A Courville Film. Visual reasoning with general conditioning layer. AAAI, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Seyedmorteza Sadat, Otmar Hilliges, and Romann Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference Learning Representations, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar
Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NewrIPS, 2022.
Christoph Schuhmann. aesth-assthetics. 2022. Accessed: 2023-11-10
Dazhong Shen, Guanglu Song, Zeyue Xue, Yun Wang, and Yu Liu. Rethinking the spatial inconsistency in classifier-free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9370-9379, 2024.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint 47X1v:240..17762, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Cei Xie, Di Chen, Feiwu Yu, and Haiming Zhao et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.203/ 14, 2025.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.093:4 2023.
12

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers arXiv preprint arXiv:2410.10629, 2024.
Wujiang Xu, Qitian Wu, Zujie Liang, Jiaojiao Han, Xuying Ning, Yunxiao Shi, Wenfang Lin, and Yongfeng Zhang. SLMRec: Distilling large language models into small for sequential recommendation. The Thirteenth International Conference on Learning Representations, 2025.
URL /RRLittpp://pperrvveeee.. net worum 2abbuArwidGGAAAR
Jiawei Yang, Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision transformers. In European Confer- ence on Computer Vision, pp. 453-469, Springer, 2024a.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint
Set::2408.060722 2024b.
Zhengqiang Zhang, Ruihuang Li, and Lei Zhang. Frecas: Efficient higher-resolution image generation via frequency-aware cascaded sampling. arXiv preprint arXiv:2410.1 /8410, 2024.
Jun Zhao, Zhihao Zhang, Yide Ma, Qi Zhang, Tao Gui, Luhui Gao, and Xuanjing Huang. Unveiling core linguistic region in large language models. arXiv preprint 1.XIv:2310.19922 2023.
Tianchen Zhao, Tongcheng Fang. Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. arXiv preprint
2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
MASSIVE ACTIVATIONS ARE THE KEY TO LOCAL DETAIL SYNTHESIS IN DIFFUSION TRANSFORMERS
A DIFFUSION TRANSFORMER ARCHITECTURE
Hidden state a;
Scale
Feedforward
Scale, Shift
Layer
Scale
Self-
Attention
Scale, Shift
Layer Norm
MLP
Hidden z
Figure 10: Illustration of the architecture of DiT block DE.-
We present the architecture of DiT block in Figure IO Each block consists of three key components: an AdaLN layer, a Self-Attention layer, and a Feedlorward layer. The AdaLN layer encodes the input timestep and the additional conditioning information (e.g., class or text embedding) into channel-wil wise scale and shift parameters ?/x and 3g. then performs Adaptive Layer Normalization (AdaLN) the hidden state zf:
* (1+ + 7x) LayerNorrmz+") + 3x.
(6)
where 7/, 38 regressed by the MLP networks of AdaLN layer conditioned on input timestep embedding and text embedding c:
7k
where ow scales the k-th residual connection.
Next, a is processed by the Self-Atiention layer to produce an intermediate feature representation. second adaptive layer normalization then applied before passing this intermediate feature the Feedforward layer, which outputs the updated hidden state. Finally, a residual connection combines the input and the transformed features produce the block output:
1++ = 2 + ag Dx(2f, ,t,
(8)
14

---

Published as a conference paper at ICLR 2026
B LAYER PROPERTIES OF MASSIVE ACTIVATIONS
In this section, we examine the layer-wise characteristics of MAs in SD3, SD3.55 and Flux. The results are shown in Figures I1 to revealing that massive activations consistently occur throughout all layers in these DiT models. Furthermore, it can be observed that the MAs dimensions remain consistent across all DiT layers (e.g.. 810 for SD3, 676 for SD3.5), as shown in Figure 14
10
Adtivation value
50-
10
proportion
10a
10-
0.1
10-
15-
20
layer
Activations along layers
Figure 11: Layer properties of MAs in SD3. Massive activations in SD3 occur all layers.
800
10a
Adiivation value.
100-
10
proportion
50-
10
10-
0.1
35
layer
Activations along layers
Figure 12: Layer properties of MAs in SD3.5. Massive activations in SD3.5 occur in all layers.
10k
10
Activatton value
1k
10
proportion
100-
10a
10-
0.1-
10
20
204
ror
50
layer
Activations along layers
Figure 13: Layer properties of MAs Flux. Massive activations in Flux occur in all layers.
Dia 810
Moan
+ Dis 676
Mean
Din 154
Dia 1446
Moan
800
Activation value
Acthotion value
10a
50
Activtton
102
20a
10a
10
15
20 layer
30
50
layer
layer
Figure 14: MA dimensions are consistent across layers. We report the average activation values of MA dimensions across DiT layers. "Mean" denotes the mean activation value of the hidden state z The results show that MA dimensions remain consistent across all layers for different DiT models (e.g., 810 for SD3, 676 for SD3.5).
15

---

Published as a conference paper at ICLR 2026
300
200
200
Activation value
Activattonn Scaling factor
Scaling value
190
Activation valum
9
Activation
280
100
.70
170
360
172
250
325
755
1000
dimension
timeseep
text
(a) Scaling factor impact activations
(b) Timestep text impact on MAs
Figure 15: Impact of the input timestep and text on Massive Activations (MAs) in SD3.5. The input timestep plays a dominant role for massive activation: the magnitude of massive activations increases steadily as decreases from 7 to O.
Activatin valu
3000
1000
Scaling factor
Scaling veluee
2000
Activatin valuee
2500
500
Activation
2003
2000
1500
154
1000
3724
1000
250
505
750
1000
dimension
timestpp
text
Scaling factor impact activations
(b) Timestep text impact on MAs
Figure 16: Impact the input timestep and text on Massive Activations (MAs) Flux. The input timestep plays a dominant role for massive activation: the magnitude of massive activations increases steadily as t decreases from to O.
Activatio value
50
Activation
Scaling value
Aciivation value
100
Activationn valuee
Scaling factor
170
170
20
160
130
140
203
114
dimersion
timestep
deess id
(a) Scaling factor impact on activations
(b) Timestep vs class impact on MAs
Figure 17: Impact of the input timestep and text on Massive Activations (MAs) in DiT-XL. The input timestep plays a dominant role for massive activation: the magnitude of massive activations increases steadily as & decreases from T/2 to O.
C TIMESTEP VS TEXT IMPACT ON MASSIVE ACTIVATIONS
We provide additional analysis for the massive activation of SD3-5, Flux and DiT-XL in Figures 17 As the hidden states 2; x+1 in DiTs computed via a residual connection (Equation 2
k+1 = x +aDD
where the ay the residual scaling factor. We first examine the impact of the residual scaling factor these activations. Specifically, we visualize the average magnitude of each dimension for activation values and scaling factor values. We observe that the residual scaling factors ag govern the dimension and values of massive activations in DìTs.
Furthermore, the residual scaling factor or is regressed by the AdaLN layer conditioned on input timestep embedding ( and text embedding
MLPx(t,c)
(10)
Therefore, we next investigate the respective impact of the input timestep and text c to the massive activations. As shown in Figure [5(b), Figure b6/b) and Figure [7]b), it can be found that the magnitude of MAs is mainly influenced by the input timestep embedding while the input text embedding exerts minimal impact it.
16

---

Published as a conference paper at ICLR 2026
D ACTIVATION INTERVENTION FOR DITS
To better understand the functional role of Massive Activations (MAs) in the visual generation of Diffusion Transformers (DiTs), we conduct an activation intervention study with four experimental settings:
Original
Non-MAs Disrupted
MAs Disrupted
Figure 18: Visual comparison activation intervention. MAs-disrupted models produce images with noticeably degraded local details, whereas non-MAs disruption preserves similar high-quality details with the original outputs. These results demonstrate that massive activations are crucial for fine-grained local detail synthesis in DiT generation process.
Original. The pretrained DIT models are used to generate visual outputs without any modification.
MAs Disrupted. We disrupt the massive activations by masking (zeroing out) their corre- sponding dimensions (e.g., dimension 293 for SD3 in Figure 2 as MAs consistently occur at fixed dimensions across all spatial tokens. Specifically. we mask the massive-actio activation dimensions in the block-output hidden state of single block and propagate the modified state through the remaining DiT blocks. All other configurations (e.g., sampling steps) are kept same to the Original setting to ensure fair comparison.
17

---

Published as a conference paper at ICLR 2026
Non-MAs Disrupted. To provide rigorous control, additionally mask equal number randomly selected non-Massive dimensions instead of the massive-activation dimensions. This setting verifies that any observed effect arises specifically from disrupting MAs rather than from the masking operation itself.
Matched-non-MAN Disrupted. To incorporate stricter control, we introduce Matched non-MAs disruption setting where the total perturbation applied non-MA dimensions is adjusted to match that of the MAs-disruption setting. Specifically, scale the masking of randomly selected non-MA dimensions so that the resulting LI perturbation norm equals that of the MA disruption. In the SD3 model, achieving this match requires applying the scaled masking to approximately 50x non-MA dimensions to reach the same perturbation magnitude of MA.
Original
Matched-non-Mi Disrupted
MAS Disrupted
Figure 19: Visualization of Matched-non-MN disruption. The Matched-non-MS. disruption preserves high-quality details similar the original outputs, but introduces uniformly distributed noise artifacts. This happens because zeroing a large set of non-MA dimensions causes a systematic shift in the hidden-state distribution, disrupting the numerical balance of the decoding process. In contrast, disrupting MAs does not introduce such noise, as MAs act as modulation signals that guide the synthesis of local details for each token. Disrupting MAs primarily suppresses the model's ability to synthesize local details without affecting the overall decoding stability.
18

---

Published as a conference paper at ICLR 2026
From the results in Figure 18 it be observed that disrupting the massive activations in DiTs markedly degrades the fidelity of fine-grained local details, whereas disrupting non-MA dimensions has almost effect the generated images. Even under the stricter Matched-non--AA, Disrupted setting (see Figure 19). the generated outputs still preserve the similar high-quality semantic structure and fine-grained detail.
In the Matched-non-MAs Disrupted setting, the generated images exhibit uniformly distributed noise artifacts. This arises because zeroing out large set of non-MA dimensions causes a systematic shift in the hidden-state distribution, which disrupts the numerical balance of the subsequent decoding process. As result, isolated noise artifacts appear in the final images. In contrast, disrupting MAs causes more focused degradation of local details without introducing such noise artifacts. This suggests that removing MAs does not alter the model's s decoding stability but instead suppresses the modulation signal responsible for synthesizing local details. The absence of noise in the MA disruption setting further emphasizes the specific role of MAs in driving local detail synthesis. These findings demonstrate that massive activations play crucial role in driving the synthesis of local details during the visual generation process DiTs.
E SPATIAL ANALYSIS OF MASSIVE ACTIVATIONS
In this section, we provide a detailed spatial analysis of the Massive Activations (MA) in DiTs.
Statistics of MA spatial map. We first analyze the statistics of the spatial map in the MA dimension (dimension 810 for SD3). As shown in Table 3 at timestep
28 15T. T, the median activation value of
the hidden state 2 is 0.60, which more than 100x smaller than the activation values in the MA dimension, where the minimum activation reaches 71.0. Furthermore, the activation values in the MA spatial map range from 71.0 to 136.0, indicating that these activations are consistently large across all spatial tokens without isolated outliers. These observations demonstrate that MAs systematically appear at a fixed dimension for all spatial tokens and exhibit similar functional behavior throughout the DiT generation process.
Spatial map of MAs: 2 [, 810] k=
=6
timestep
Median(=
Min
Max
Mean
Std
23
72.0
135.0
87.4
7.6
0.57
f
71.0
136.0
89.5
9.6
0.60
28
t
70.0
138.0
86.4
8.8
0.60
28
Table 3: Statistics of MA spatial map for SD3.
100
140
SD23
Mlcck
88
86
70
28
140
170
SD3355
soock i
135
92
88
98
30
30
Honwwdeg
Figure 20: Visualizations MA spatial map: MAs appear across all spatial tokens, while tokens corresponding to detail-richer regions (e.g.. the fox) exhibit slightly higher MA values.
Spatial map of MA dimension. To analyze the spatial characteristics MA, we visualize the spatial maps of MA dimension across different timesteps, as shown in Figure 20 From the results, we obtain
19

---

Published as a conference paper at ICLR 2026
the following observations: (1) Activations in the MA dimension remain consistently large across all spatial locations. For SD3.5, the MA values range from approximately 70 to 180 without any extreme outliers. These activation values more than 100x larger than the median activation value of the hidden states (approximately 0.6 from Table 3p. This confirms that MAs appear across all spatial tokens, which serves token-wise modulation signal for local detail synthesis of each token. (2) Tokens corresponding to detail-richer regions exhibit slightly higher MA values. In particular, the region containing the fox shows marginally higher MA values. This suggests that while the MA dimension is activated for every spatial token, its magnitude adapts subtly to local visual complexity. Spatial tokens responsible for synthesizing richer or more intricate details receive slightly stronger MA responses.
F ADAPTING DG TO OTHER SCHEMES
Our DG strategy a detail guidance approach driven by massive activations, and it is applicable to wide range of models that exhibit massive activations, not limited to the standard DiTs. To assess the generality of our approach across different architectures, we evaluate our strategy on several other models, including efficient DiT' and non-DiT models.
Efficient DIT. To examine the robustness of DG on efficient DīT, we conduct experiments with the popular PixArttalpha (Chen et al.) 2023). PixArt-alpha adopts an efficient DiT architecture that injects text conditioning through cross-aftention and employs shared AdaLN layers across all blocks to encode the timestep. These experiments allow us to specifically verify whether DG remains applicable under the shared-AdaLN setting.
Non-DiT. To further examine DG beyond transformer-basee diffusion architectures, we experiment with the recent SANA (Xie et al. 2024), which introduces a novel Linear-DiT design. SANA replaces all vanilla attention with linear attention and integrates 3x3 convolutions into the MLP. These experiments allow us to verify whether DG remains effective under such a hybrid design that incorporates convolutional operations
Implementation details. We analyze the pattern of Massive Activations (MAs) in the models and observe prominent MAs phenomena in both Pix/rrt-alpha and SANA. These models exhibit similar characteristics to standard DiTs, with MAS primarily occurring across all spatial tokens of fixed dimensions. For PixArt-alpha, we create "bad" version by masking dimension 273 in block 6. In the case of SANA, we mask dimensions 56 and 597 in block 2 to disrupt the MAs.
Prompt Alignment
Detail Quality
Model
Type
DG
Clipscore
Blipscore
HPSv2.1
Aesthetic
22.64
68.41
25.63
6.01
Cond
V
23.43
72.07
29.18
6.53
PixArt-alpha
X
26.20
87.64
29.99
6.21
CFG
V
26.17
86.88
30.74
6.34
X
23.52
78.25
24.40
5.91
Cond
V
24.98
84.11
28.72
6.12
SANA
27.07
91.03
30.13
6.00
CFG
27.20
90.25
30.52
6.07
Table 4: Evaluation of DG other architectures. Our DG strategy generalizes effectively to other architectures and significantly enhances the visual detail quality for both Pixart-Alpha and SANA.
As shown in Table 4 our DG strategy consistently improves the performance of both PixArt-alpha and SANA. In the conditional generation setting, DG yields substantial improvements in detail quality, HPSv2.1 increases from 24.40 to 28.72 and the Aesthetic score from 5.91 to 6.12 2 for the Linear-DiT SANA model. Moreover, when combined with CFG, DG further enhances visual quality, improving HPSv2. from 30.13 to 30.52 and the Aesthetic score from 6.00 to 6.07. These results indicate that DG significantly boosts visual fidelity, with particularly notable gains in fine- mne-grained local details. Overall, these findings demonstrate that DG generalizes effectively beyond standard DiT architectures, delivering robust improvements across both efficient DiT variants and non-Di non-DiT models.
20

---

Published as a conference paper at ICLR 2026
G COMPUTATIONAL OVERHEAD OF DG
In this section, we investigate the computational overhead of our DG strategy. Specifically, we generate 100 images 1024x1024 resolution using single L40S (48GB) GPU. The average computational cost is reported in T
Model
Type
GPU memory (GB)
Generation latency (8)
Aesthetic
Cond
17
2.3
5.58
CFG
20
4.3
5.80
SD3
DG (Ours)
20
3.5
6.01
Cond
28
7.2
5.94
CFG
32
15.7
6.01
SD3.5
DG (Ours)
32
10.6
6.16
Cond
35
16.2
5,50
CFG
42
36.0
5.96
Flux
DG (Ours)
42
24.8
6.13
Table 5: DG is approximately 1.5x faster than CFG. Since DG only forwards one conditional branch before the disrupted depth m, while CFG requires both the conditional and unconditional branches to be processed, DG achieves superior efficiency. For the SD3.5 model, our DG generates 1024x1024 image in 10.6s, approximately 1.5x faster than CFG, which takes 15.7s.
DG is approximately 1.5x faster than CFG and delivers superior performance. As demonstrated in Table 5) DG achieves higher performance than CFG (e.g., 6.16 vs. 6.01 on SD3.5 model). Moreover, DG generates a 1024x1024 image in 10.6s, approximately 1.5x faster than CFG, which requires 15.7s. This efficiency stems from DG's architectural design: it leverages an MAs-disrupted conditional branch to guide the base model. Before the disruption depth (e.g. m blocks for SD3.5), DG requires forwarding only the conditional branch, while CFG necessitates forwarding both the conditional and unconditional branches throughout the entire sampling process. This architectural difference makes DG both more efficient and effective than CFG.
out of 38
H MORE IMPLEMENTATION DETAILS
H.1 IMPLEMENTATION DETAILS FOR FIGURES
This section provides the implementation details the figures 1-5 presented in our paper.
Figure 1. Prompts used for the visual examples in the figure:
Prompts for the examples with Detail Guidance (DG), [(1) "Card Magic the gathering style of tom whalen022 2e SM Ricardo. Lavage du (2) "Movie Still of The Joker
char au gazole a Biesheim." ", wielding red Lightsaber, Darth Joker sinister evil clown prince of crime, HD Photograph",
(3)"Bichon maltais fou", "F)nk Lloyd public library with coffee shop, mid century, interior"].
Prompts for the examples with DG and CFG. [(1) cat, chubby, very fine wispy and extremlly long swirly wavy fur, under water, Kuniyoshi Utagawa, Hishida Shunso, a very curvy chubby cat, golden embroidery fabric kimono, flowing glowing biomorphic wisps, phosphorescent swirls, tendrils, wavelets, streamers, a murmuration of bioluminescent bubbles, detailed and intricate, elegant aesthetic, ornate, finely detailed, 128K UHD Unreal Engine, octane, fractal pi, fBm", (D)"Dream alpine treehouse with sweeping mountain views")
Figures to 5. For configuration of the hidden states (block index k and timestep index t) used for different DIT models, please refer to Table
21

---

Published as a conference paper at ICLR 2026
Figure 2
Figure 3
Figure 4
Figure 5
D/T-XL/2
SD3
SD3.5
Flux
Architecture
Image size
256x256
1024x 1024
256x256
1024x1024
1024x1024
Hidden size
1152
1536
2432
3072
1536
1536
Hidden state 2;
Total blocks N
28
24
38
57
24
24
Block index
14
12
19
28
12
Total timesteps T
250
28
28
50
250
28
28
Timestep index
125
14
14
25
125
14
Table 6: Configuration of hidden states used in Figures 2-5.
H.2 IMPLEMENTATION DETAILS OF EXPERIMENTAL RESULTS
We implement DG three pretrained large diffusion models: SD3-Medium (Esser al. 2024).
SD3.5-Large (Esser et al. 2024), and Flux-dev bblack-forest labs 2024). Notably, Flux is CFG-distilled model. To evaluate DG independently of CFG, we adopt the de-distilled variant
from (black-forest labs 2024). Full experimental settings are provided in Tables [ndd
Configurations for DG. For each Diffusion Transformer, construct degraded detail-deficient model Do.m by disrupting the dimensions corresponding to massive activations in the m-th blocks, following the intrinsic activation patterns of each DIT. Detailed configurations are summarized in Table 7
Hyperparameters setup. All models adopt the default diffusion sampling settings (e.g., sampler type and number of steps). Specific hyperparameter choices are listed in Table g
Computing Resources. All experiments are performed on single NVIDIA L40S (48 GB) GPU. DG builds the degraded detail-deficient model by directly disrupting massive activations in hidden states without additional training.
Model
Blocks N
Hidden size d
Disrupted dimensions
Disrupted depth m
SD3
24
1536
810
6
SD3.5
38
2432
676
20
Flux
57
3072
[154, 1446]
22
Table Configurations Detail Guidance (DG) for different DiTs.
Model
Guidance Type
sampling step
À
CFG
28
SD3
DG
28
1
CFG+DG
28
3
CFG
28
4
SD3-5
DG
28
4
CFG+DG
28
2
CFG
50
3.5
Flux
DG
50
4
CFG+DG
50
3
2
Table 8: Hyperparameter setup.
22

---

Published as a conference paper at ICLR 2026
H.3 EVALUATION DETAILS
We evaluate different guidance strategies from two key perspectives: prompr alignment and detail quality. Prompt alignment reflects how well the generated image semantically matches the input prompt, while detail quality measures the fidelity and richness of fine-grained visual details.
Specifically, we adopt Clipscore (Radford et al. 2021) and Blipscore (Li al., 2022) to quantify prompt alignment, and employ HPSv2.1 (Wu et al. (023) and Laion-Aesthetics (Schuhmann 2022)
as indicators of visual detail quality. The details of each metric are as follows.
Clipscore measures the global semantic consistency between text and image by computing the cosine similarity between their CLIP-encoded features. We adopt the hhip-vitlargee-atchhe version all experiments.
Blipscore estimates prompt-image alignment through fine-grained image-text matching model (BLIP), capturing nuanced semantic relationships beyond global similarity. We use the blip-itm-large- coco version to evaluate the model.
HPSv2.1 is human preference score that provides perceptual measure of visual realism and aesthetic quality. is widely used to benchmark high-fidee fideliid image synthesis, and we adopt HPSv2.1 for evaluation.
Laion-Aesthetics predicts aesthetic appeal using a model trained on LAION's large-scale human- rated dataset, serving automated proxy for human aesthetic assessment.
I CLASS-CONDITIONAL GENERATION
To evaluate the robustness of our Detail Guidance (DG) strategy, we perform class-conditional gener- ation the ImageNet 256x256 dataset by applying DG the pretrained DiT-XI1/2 model (Peebles)
Xie 2023).
Type
DG
FID
f
Prec. t
Rec. t
16.95
105.64
0.61
0.76
Uncond
9.68
122.22
0.66
0.67
9.52
122.79
0.66
0.63
Cond
5.77
179.26
0.78
0.55
Table Performance comparison on dataset ImageNet 256
256. Prec: Precision, Rec: Recall.
For DIT-XL/2, we set the disrupted depth 7. We assess DG under both unconditional and conditional generation settings, with results reported in Table DG delivers consistent performance improvements in both settings, demonstrating the robustness of our guidance strategy.
USER STUDY
SD3
SD3.5
Flax
CFG-DG (Ours)
CFG-DG (Ours)
CFG-DG (Ours)
80
78 54
31.48
80
75-53
7%
62.51
65.31
60
48.71
46.5 54
40
1114
40
40
20
21.46
20
24,7
20
Prompt
Color
Detail
Prompt
Color
Detail
Prompt
Color
Detail
Figure 21: User study on SD3, SD3.5, and Flux. We report the win rates comparing CFG with method.
We conduct a user study to evaluate the benefits of our DG strategy from three key aspects: prompt alignment, detail preservation, and color consistency. Prompt alignment measures how well the
23

---

Published as a conference paper at ICLR 2026
generated images match the imput prompts. Detail preservation reflects the fidelity of fine- fine-grained visual details, while color consistency captures the naturalness and realism of colors.
For each model, 20 annotators compared 100 pairs of images produced by CFG and CFG DG with respect to these criteria. We report the averaged win rates in Figure [T] which show that our approach yields significant improvements across all metrics, particularly for Detail and Color.
K DISCUSSION: CFG AND AUTOGUIDANCE VS. OUR DG
To facilitate a clearer conceptual understanding of DG and its relationship to existing guidance strategies, we present unified formulation of CFG, Autoguidance, and DG in Table TO
Type
Formulation
Disruption
CFG
Do (zi, (z, (,c) Ds Do (2) (Ds (,), De (zi, t)) Do (,)) = Dy (2,,, c) + w (Da (2,,,,) - Dy- (ze,t, Do (2, t.c) = (2, () (D, (2), - Du (2,,
Autoguidance
DG
2x
Table 10: Unified formulations of CFG, Autoguidance and Our DG. é denotes the unconditional prompt, 0 denotes under-capability model, and & denotes with MAs-disrupted hidden state.
Under this unified formulation (see Table 10 the distinction becomes clear:
CFG constructs a degraded version by "disrupting" the input prompt condition c, resulting in prompt-alignment guidance.
Autoguidance constructs degraded version by "disrupting" the model 0, resulting in entangled prompt-aligament and visual detail guidance.
Our DG constructs degraded version by "disrupting" the visual input 2; itself through MA disruption, resulting in visual detail guidance.
Our DG can be seamlessly combined with CFG, enabling a decoupled guidance mechanism where CFG handles prompt alignment while DG focuses visual detail guidance.
L ANALYSIS OF FAILURE CASES
In this section, we present visualizations and analyses of the failure cases associated with our DG strategy. DG explicitly designed enhance the local detail fidelity of generated images. However, this emphasis on fine ine-grained detail can occasionally compromise semantic faithfulness. In particular, when the prompt specifies strong stylistic, identity-related, or conceptual requirements, DG may favor detail enhancement over strict adherence to the intended semantics.
As illustrated Figure 22] DG produces outputs with noticeably richer textures and enhanced local details but may fail to fully satisfy the semantic requirements of the prompt. For instance, given the prompt "The Mona Lisa as a vogue model, 1989 punk-inspired portrait, dramatic lighting, cinematic lighting". ", DG generates portrait with improved local details and textures, yet the output lacks the expected vogue style and the distinctive identity features of Mona Lisa. A promising direction to address this limitation is to combine DG with CFG, enabling joint control over both local detail and semantic alignment. Moreover, when the guidance scale becomes relatively large (e.g., greater than 5), our DG strategy may also exhibit oversaturation effects, similar those observed with CFG.
M LIMITATIONS AND FUTURE WORK
Limitations. Our Detail Guidance (DG) method is primarily designed to explicitly enhance the detail quality of generated images. While does improve prompt alignment, its ability to ensure strong aligmment throughout the generation process is somewhat constrained. For prompts that specify strong stylistic, identity-rllated or conceptual requirements, we recommend combining DG with CFG to provide joint guidance on prompt alignment and detail enhancement.
24

---

Published as a conference paper at ICLR 2026
Prompt
DG(Ours)
CFG
20 young girl cyberpunk outfit
Flat design eye icon
The Mona Lísa as a vogue model, 1989 bunk-inspired portrait, dramatic lighting, cinematic lighting
Figure 22: Failure Cases of Our DG. DG explicitly enhances local detail fidelity, which can occasionally compromise semantic alignment. DG produces outputs with improved texture and fine-grained details but fails to satisfy the prompt requirement (c.g., "Mona Lisa as a Vogue model"), leading missing identity and stylistic attributes. The words highlighted in red correspond to semantic requirements the prompt that are not fulfilled by the generated image from DG.
This paper mainly explores the potential of Massive Activations (MAs) during the sampling stage of DiT'. A more promising direction for future work lies in leveraging the capacity of MAs during the training stage. Since MAS play a crucial role in local detail synthesis, incorporating them more effectively into the training process could significantly enhance the performance of DiTs. This approach offers valuable insights for optimizing future DiTs, driving further advancements in their generative capabilities.
Future Work.
N USE LARGE LANGUAGE MODELS (LLMs)
In preparing this manuscript, we used large language models solely as lightweight writing aid for grammar, wording, and formatting suggestions. The models were not used to generate research ideas, design algorithms, write code, experiments, analyze data, or draft scientific content. All technical claims, methods, and conclusions were conceived, produced, and verified by the authors. Suggested edits from LLMs
manually reviewed and integrated at the authors' discretion. And we accept full responsibility for the accuracy and integrity the manuscript, including ensuring that no plagiarized or misrepresented content from LLM included.
25

---

Published as a conference paper at ICLR 2026
O
MORE QUALITATIVE RESULTS FOR INTEGRATION WITH CFG
lo
lol
CFC
lol
DG
lol
CFG
Figure 23: Visual results SD3.
CFC
CFC + DG
Figure 24: Visual results on SD3.
CFC
CFC + DG
Figure 25; Visual results on SD3.5.
26

---

Published as a conference paper at ICLR 2026
P
MORE QUALITATIVE RESULTS FOR DG
Baseline
CFG
(G()
spanish water dog breed as arthur morgan from red dead redemption
cherry-colored dream, a fantasy painting
anime portrait of beautiful vamire witch, sci fi suit, intricate, highly detailed
mouse
An image of cyborg
Figure 26: Visual results on SD3. Baseline indicates visual output without CFG.
27

---

Published as a conference paper at ICLR 2026
Baseline
CFG
DC(our)
swirling water tornados epic fantasy
cinematic still of highly reflective stainless steel train in the desert, at sunset
Hatasine Miki ess
Hatsune MiKur. es real
Hatsune Miku es real
sign that says "Hatsune Miku es real"
little bunny
Most Expensive Sports Car
Figure 27: Visual results on SD3.5. Baseline indicates visual output without CFG.
28

---

Published as a conference paper at ICLR 2026
Baseline
CFC
DG(ours)
A close-up photograph of fat orange cat with lasagna in its mouth
toilet design toilet in style of dodge charger toilet, black
aristocratic russian noblewoman, dressed in medieval dress, model Face
Movie Still of The Joker wielding red Lightsaber
a playful maid, undercut hair, apron, amazing body, pronounced feminine Feature
Figure 28: Visual results on Flux. Baseline indicates visual output without CFG.
29


Reviews:
[{"review_id": "Zs5g2jamPm", "reviewer": "Reviewer_sJzY", "summary": "This paper systematically studies the role of Massive Activations in image generation and discovers that they play a crucial role in generating fine-grained local details. Building on this observation, the authors propose a simple, training-free method that constructs a “detail-deficient” model by deliberately disrupting these Massive Activations. This degraded model is then used as a negative reference to guide the original network (similar to CFG) to generating images with better detail fidelity.", "questions": "- Can it be adapted to non-DiT or hybrid architectures?\n- What is the computational overhead and inference latency of DG?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and well-structured. It first provide a clear analysis of Massive Activations in Diffusion Transformers, and then proposes a simple and effective method based on this observation to improve image generation quality.\n\n- The proposed approach is simple and training-free. It only involves disrupting the massive activations within a pretrained model to construct a degraded variant, which is then used as a negative reference through a CFG-style guidance to enhance fine-grained visual details.\n\n- Strong results. The proposed method shows strong performance gains and visual improvements", "weaknesses": "- DG relies on fixed-dimension activation patterns and the AdaLN scaling mechanism specific to DiTs. It is unclear whether the approach generalizes to non-transformer or hybrid architectures. Moreover, several recent works have suggested that AdaLN may not be the most optimal solution due to its parameter overhead and have proposed lighter alternatives. It would be valuable to discuss whether DG can be adapted to other schemes.\n\n- No ablation on computational overhead or inference latency. Since the proposed method requires constructing and utilizing a degraded model, it would be helpful to provide a comparison of GPU memory usage and inference time with and without DG to better understand its efficiency trade-offs.\n\n\n- It is good to also discuss the failure cases and limitations for the method. For example, while the finding that disrupting MAs mainly affects local details rather than semantics is compelling, such disruptions could also introduce unintended side effects. For exmaple I am gussing over-sharpening or loss of texture consistency. Additional experiments or qualitative examples discussing limitations would make the paper more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "67s76rWYLR", "reviewer": "Reviewer_RLWh", "summary": "This paper systematically studies the massive activations in the DiT (Diffusion Transformer) image generation framework. The authors find that these activations are highly correlated with detail synthesis. Based on this observation, they propose a guidance (DG) strategy to steer the generation process toward better texture synthesis. Experiments demonstrate its effectiveness.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The presentation is well-organized and easy to follow.\n2. The study on massive activations is detailed and systematic.\n3. The proposed DG strategy is effective and novel.", "weaknesses": "1. The paper lacks comparison with advanced CFG strategies, such as PAG [1], FA-CFG [2], and Semantic-CFG [3]. While such comparisons are not essential, including them would strengthen the experimental validation and enhance the robustness of the evaluation if avaialbe.\n\n2. In Figure 2, the authors claim that massive activations appear at fixed dimensions across all patch tokens. Is this dimension consistent across all layers?\n\n[1] Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance.\n[2] FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling.\n[3] Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p9MaTe2fSq", "reviewer": "Reviewer_SiyV", "summary": "The authors study the role of massive activation (MA) in DiT models. Massive activation is a common observation in transformer models. The authors attribute massive activations in DiT to the generation of local detail. Experiments show that disruption of MA degrades local information. The authors propose Detail Guidance (DG) to enhance CFG on local details. The manuscript is clearly written. The analysis of MA is comprehension. The experimental design is legit. Overall, it is a high-quality study. Though, I note several unclear parts below and suggest several controls to support the effectiveness of DG.", "questions": "**Spatial map of MA**\n\nIn the ViT studies, the MA often appears in only several background tokens in the image. If the role of MA in DiT is to synthesize details, then does it mean the MA is supposed to be in all the tokens, or only in tokens with plenty of details? If only disrupting several tokens’ MA, is the details only missing locally? The special activation map of MA dim is not shown. Showing an activation map of MA dim would be helpful to understand its actual role.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Massive activation is a common observation in transformer models across modalities. Its functional role is less known in DiT models. The analysis provided by the study is comprehensive (Figure 3-5). The finding of MA’s role of synthesizing is novel and supported. Leveraging this finding, Detail Guidance seems to be an effective, though simple, way to enhance local details.", "weaknesses": "**DG control**\n\nAs the authors mentioned, Karras et al 2024 proposed using an under-trained version to contrast the condition path. DG works in a similar way. Method in Karras et al 2024 should be used as a control. Or, if MA-disruption is really only about disrupting local detail features, there should be another control that uses blurred conditional path as the unconditional signal.\n\n**MA-disruption control**\n\nIn “Non-MA disruption”, it is hard to say that zeroing non-MA dims is a fair control. The amount of perturbation in “Non-MA disruption” is smaller than in “MA disruption”. To be a fair control, the same total magnitude of disruption should be the same.\n\n**Missing information, or not emphasized in the main text**\n\nAre MA studied in the conditional path (D_/theta(z,t,c) in eq 1) or unconditional path (D_/theta(z,t) in eq 1)? From eq. 2, it seems the state is taken from the unconditional path, but in eq 4, the MA is studied/manipulated in the conditional path. The interpretation of the results heavily depends on where MA are found and manipulated. Authors should clarify where the hidden states are taken from in Figure 2-5.\n\nFigure 2 misses information on which layer and time step the activations are taken from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)