Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models
Authors: Donghoon Ahn
Jiwon Kang
Sanghyun Lee
Minjae Kim
Wooseok Jang
Jaewon Min
Sangwu Lee
Sayak Paul
Seungryong Kim
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
flow matching models
diffusion guidance
generative models
Primary area: applications
TL;DR: We analyze head-level attention perturbation guidance to improving image quality and style in diffusion and flow model, additionally introducing a generalized attention perturbation framework.

Abstract: Recent guidance methods in diffusion models steer reverse sampling by perturbing the model to construct an implicit weak model and guide generation away from it. Among these approaches, attention perturbation has demonstrated strong empirical performance in unconditional scenarios where classifier-free guidance is not applicable. However, existing attention perturbation methods lack principled approaches for determining where perturbations should be applied, particularly in Diffusion Transformer (DiT) architectures where quality-relevant computations are distributed across layers. In this paper, we investigate the granularity of attention perturbations, ranging from the layer level down to individual attention heads, and discover that specific heads govern distinct visual concepts such as structure, style, and texture quality. Building on this insight, we propose ``HeadHunter", a systematic framework for iteratively selecting attention heads that align with user-centric objectives, enabling fine-grained control over generation quality and visual attributes. In addition, we introduce SoftPAG, which linearly interpolates each selected head’s attention map toward an identity matrix, providing a continuous knob to tune perturbation strength and suppress artifacts. Our approach not only mitigates the oversmoothing issues of existing layer-level perturbation but also enables targeted manipulation of specific visual styles through compositional head selection. We validate our method on modern large-scale DiT-based text-to-image models including Stable Diffusion 3 and FLUX.1, demonstrating superior performance in both general quality enhancement and style-specific guidance. Our work provides the first head-level analysis of attention perturbation in diffusion models, uncovering interpretable specialization within attention layers and enabling practical design of effective perturbation strategies.

Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models
Donghoon Ahn'
Jiwon Kang*
Sanghyun Lee
Minjae Kim
Jaewon Min
Wooseok Jang'
Sangwu Lee
Sayak Paul
Seungryong Kim
KAIST AI
2 Korea University
KKeea AI
Huuggin Face
Abstract
Recent guidance methods diffusion models steer reverse sampling by perturbing the model construct an implicit weak model and guide generation away from it. Among these approaches, attention perturbation has demonstrated strong em- pirical performance in unconditional scenarios where classifier-free guidance is not applicable. However, existing attention perturbation methods lack principled approaches for determining where perturbations should be applied, particularly in Diffusion Transformer (DiT) architectures where quality-relevant computations are distributed across layers. this paper, we investigate the granularity of attention perturbations, ranging from the layer level down to individual attention heads, and discover that specific heads govern distinct visual concepts such as structure, style. and texture quality. Building on this insight, we propose "HeadHunter" a systematic framework for iteratively selecting attention heads that align with user-centric objectives, enabling fine-grained control over generation quality and visual attributes. In addition, we introduce SoftPAG, which linearly interpolates each selected head's attention map toward an identity matrix, providing continu- ous knob to tune perturbation strength and suppress artifacts. Our approach not only mitigates the oversmoothing issues of existing layer-level perturbation but also enables targeted manipulation of specific visual styles through compositional head selection. We validate method modern large-scale DiT-based text-to- image models including Stable Diffusion 3 and FLUX.1, demonstrating superior performance in both general quality enhancement and style-specific guidance. Our work provides the first head-level analysis of attention perturbation in diffusion models, uncovering interpretable specialization within attention layers and enabling practical design of effective perturbation strategies. Our project page is available available at http:://aaxv.oop/aa///500..11111
The latest version of this paper is
Introduction
Diffusion models 19 54 56 58
[1
45 42
and flow-matching models 34 37 [4] have gained
great popularity in visual generation tasks, including images [45 42 5 14), videos 21 [8] 3 63), 3D
43 32 60 50), and 4D content [5T 64 61]. The key behind the success of diffusion models lies in classifier-free guidance (CFG) 20 which substantially enhances image generation quality during conditional inference. Despite its effectiveness, CFG has two key limitations. First, applies only conditional generation, limiting its applicability in unconditional settings such inverse
Equally contributed firs author, o: Equally contributed as second author, t: Correspondngg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

problems [25 8 55 9 Second, CFG often reduces sample diversity and leads to over-saturated or overly simplified outputs
48 26
To address the limitations of CFG in unconditional scenarios, alternative guidance strategies [ 23
22
have been proposed. These methods steer the denoising trajectory by perturbing the input or the model itself, or by training weaker model, thereby guiding samples away from low- quality regions and toward the high-quality data manifold. Among these strategies, attention-layer perturbation approaches [ [22] continue to be a practical and widely explored strategy, as they can generate well-aligned weak models without additional training. Some papers explain these guidance methods from an energy perspective [22] and draw connections to Hopfield networks [T0] [44, which empirically work well with specific blocks (medium blocks of U-Net [46] architecture) of attention.
26 24
However, there is still limited understanding of where perturbations should be applied. In particular, Diffusion Transformer (DiT) [4] architectures lack localized blocks responsible for global semantics, unlike U-Nets [46), and instead distribute this functionality more evenly across all layers (2]. This structural difference makes it even more critical carefully select perturbation targets achieve effective guidance.
To determine suitable perturbation targets, we first consider the underlying structure of DiT. In DiT, multi-heas self-attention [59] plays a central role in modeling global dependencies. Each head attends to different aspects of the input, and prior work has shown that heads often specialize in distinet semantic features [59 40 12 [6]. This suggests that attention heads, rather than entire layers, may serve more precise and effective targets for perturbation.
Motivated by this, we explore finer-grained yet semantically meaningful computational units within attention layers: attention heads. Interestingly, we observe that individual head-level perturbation guidance often capture interpretable visual concepts and specialize in tasks such as enhancing structural fidelity or injecting stylistic elements. Moreover, these functional roles can be composed by combining multiple heads.
Building on these observations, we (i) analyze key properties of head-level perturbation guidance, including composability and controllability, (ii) propose HeadHunter, systematic framework for retrieving heads that align with arbitrary objectives, and (iii) introduce SoftPAG, a variant of PAG [ that enables fine-gran fine-grained, continuous control over guidance strength, mitigating over-smoothness and oversimplification caused by overly aggressive perturbations.
We demonstrate that HeadHunter not only outperforms layer-level perturbation guidance in general quality improvement but also enables targeted manipulation of visual styles, as supported by strong qualitative and quantitative results.
In summary, our main contributions are as follows:
To the best of our knowledge, we are the first to apply perturbations at the level of individual attention heads, enabling fine-grained and concept-specific control.
We analyze the properties of head-level perturbation guidance in Diffusion Transformers (DiT), providing insights into the specialization and combination of such heads.
We propose HeadHunter, a systematic head selection framework for arbitrary objectives, and demonstrate its effectiveness in both general quality enhancement and style-specific guidance.
We introduce SoftPAG, which interpolates each selected head's attention map toward the identity matrix, providing continuous knob to tune perturbation strength and to mitigate oversaturation and oversimplification
2 Preliminaries
Diffusion and flow matching models. Diffusion models 19 52 57 define a generative process as iterative denoising of sample drawn from a Gaussian prior. The forward process gradually perturbs clean data point Xy Pumaa through the noise schedule (a,, ot). as x, = a2Xy + oyc, where
~ (O I) and E [0, 1]. A neural network čo (x, t) learns to predict the added noise c, enabling
reverse-time sampling from noise to data. On the other hand, flow matching [35] 36 provides a
deterministic alternative by learning continuous-time velocity field that guides a linear interpolation
from noise to data: X, = (1 1)xo + l,e, where e ~ NN I), and the associated velocity field
u(x, t)
Xo. A neural network úg(X,, t) is trained to approximate this target velocity.

---

Multi-head attention mechanism. Vaswani et al. [59] introduced the multi-head attention mech- anism, which projects queries, keys, and values into multiple subspaces and computes attention in parallel. At the 1-th layer of the denoising network ég, each head h E (I,..., Hi) applies distinct linear projections:
- Q. 1,6 K,,. K.W VLA = V,W 1,b
where Q K, VI E R Xx are the query, key, and value matrices, WQ.KV E Rd are the head- 1,b
specific projections, d is the per-head dimensionality, and H2 is the number of attention heads in layer 1. The attention map and output for each head are
QuAKin vd
A = Softmax
OLA = AL Vibs
(2)
and all head outputs are concatenated and projected:
MultiHead( Qi K2, Vi) Concat Concat(O..,,-.. (0,,,..., OI,H,) WW:
where W E (Hdl) is the output projection matrix. Multi-head attention enables the model to capture diverse semantic relations across tokens
Classifier-free guidance. Classifier- free guidance (CFG) (20 enhances conditional generation by extrapolating predictions from conditional and unconditional models:
êCTG (1 w &oond
W'unnoond
where w is the guidance scale, and écond, Euncond denote the predicted noise with and without condi- tioning, respectively.
Attention perturbation guidance. As attention maps encode spatial and semantic structures 39 perturbing them can guide the denoising trajectory away from suboptimal predictions, improving structural fidelity at test time [1 [22 26 Analogous to classifier-free guidance (20 attention
perturbation guidance extrapolates between original and perturbed predictions:
Éguided + w) Eoriginal pertturreed
where éperturbed is obtained by modifying attention maps AI, in Eq. during the forward pass.
Let L denote the perturbed layers and Hi the number of heads layer I. Perturbations are applied
to Ar for all heads h E {1...., Hi} and E L. In perturbed-attention guidance (PAG) (0.. the attention maps are replaced with the identity matrix, disabling contextual aggregation:
A (PAG) Ak
I E R NxN
3 Motivation
Figure Motivating example. Each image generated with PAG I where perturbation is applied single attention head within DiTss. Guiding with different perturbed attention heads produces notably distinct results. Results for additional heads are provided in Appendix [I.]] All images are generated with the prompt "smiling girl holding cat, in flower garden" " using wtteeeaafffffii------eee-- perturbed across columns.
Each row corresponds to single layer, with different heads
steers generation away from weaker predictions by slightly altering the model's forward pass. Thus, deciding what weaken becomes critical, but principled methods for determining where in the diffusion network to apply perturbation remain underexplored. As result, prior works often rely on heuristic selection of perturbation targets, such
Attention perturbation guidance 1
26

---

Layer-leveel guidance
Head-Ievel guidance
L2
L22
L
L13
(4) Layer-level
(b) Head-level perturbation guidance in each layer
(c) Using top- performing heads
perturbation guidance
Figure 2: Generated images from head- and layer-level perturbation guidance. (a) Results
of layer-level perturbation guidance, where perturbation applied to all heads in the layer. (b)
Results of head-level perturbation guidance, where each result is obtained by independently applying perturbation to a single head of the layer. Red boxes indicate high-performing heads in terms of
PickScore [27] (c) Perturbation guidance using only the high-performing heads identified in (b) yields higher-quality generations across both low-performing layers (L13) and well-performing ones (L.12). The prompt "Turkish girl with lantern, dark room" is used.
as the middle block of a U-Net [46], which is known to process high-level semantic information with global self-attention 23
However, unlike U-Net, transformer-based architectures as in DiTs 41
15
lack a coarse-
to-fine synthesis structure (2. This absence of an explicit bottleneck makes it challenging directly identify layers responsible for high-level semantics. Instead, the model distributes semantic processing more uniformly across layers. In this setting, attention, particularly multi-hear self-attention [59]. plays central role in modeling global dependencies without convolutions.
Each attention head processes different aspects of the input, and prior works in large language models and vision transformers have shown that heads specialize in capturing distinct semantic attributes [59 40 12, 16]. This suggests that attention heads, rather than entire layers, may serve as more effective units for applying perturbation.
Motivated by this observation, we increase the granularity of attention perturbation from entire layers to individual attention heads. In conventional approaches, referred as layer-level perturbation guidance (or simply layer-level guidance), perturbations are applied uniformly to all attention heads within selected layers. In contrast, our proposed head-level perturbation guidance (or simply head- level guidance) selectively perturbs specific attention heads, enabling finer and more targeted control. As shown in Fig. perturbing different heads leads to clearly distinct effects in the generated images, indicating that heads act semantically meaningful substructures. This highlights the limitations of layer-level perturbation, which may be overly coarse and suboptimal. Even with carefully chosen layers, layer-level perturbation fails to leverage the modularity and functional diversity inherent in multi-head attention, motivating our focus on head-level guidance.
3.1 Analysis head-level perturbation guidance
Definition. We define head-level perturbation guidance applying perturbations selectively to sub- set of attention heads during the forward pass. Specifically, given set S {(4,, hi)........ (lms hmm) )
of m selected (layer, head) pairs, replace their attention maps with identity matrices:
A (PAG) LA
= for (L, E S,
where A, (PAG) I.b denotes the perturbed attention map of head h at layer I, and I is the identity matrix used in PAG M While we illustrate this with PAG, the formulation also applies to other perturbation methods such as SEG [22]. with each head perturbed independently. See Appendix F for details.
Inefficiency of layer-level guidance due to intra-layer diversity. Even when we perturb the attention map of a single layer within diffusion models, exhibits highly polysemantic attentive be- havior, as the computation in individual heads occurs independently and diversely [13 40 Therefore, applying the same perturbations across heads in layer fails to account for this intra-layer diversity, potentially disrupting the output from the guidance and leading to unintended side effects. As shown

---

H10
Dark
L11 H15
Dark Shourang
Stewring
L13 H17
Shadow
H23
Bine
(a) Individual heads
(b) Multiple heads
Figure 3: Effeet of head-level guidance on concept amplification and combination. (a) Guiding with individual heads amplifies specific visual concepts such darkness, geometry, shadow, color. (b) Guiding with two heads simultaneously combines their effects in the output.
in Fig. 2. some heads from supposedly poor layers (L13 of Fig. 2 (a)) still produce high-quality results (Fig. (b)). This suggests that overall quality may degrade due the effects of a few heads dominating the outcome, indicating further perturbed only high-scoring heads based on preference scores. The resulting samples (Fig. (c)) show clear improvements standard layer-level guidance. These results demonstrate the inefficiency of coarse layer-level guidance and motivate us to analyze head-level guidance and its properties.
for improvement. To test this, filtered and
Individual head-level guidance occasionally reveals interpretable concepts. As can be seen in Fig. [ we observe that perturbing individual heads for guidance leads to highly diverse behaviors in terms of their influence on generation. Beyond quality, we find that certain heads amplify specific visual attributes when used for guidance. These include texture, color tone, geometry, and lighting. We provide some examples Fig. 3 (L11 (L11,H15 H15) consistently induces shearing, while other heads amplify darkness or blue tone. Additional interpretable heads and their effects are shown in Appendix [2.2 We further analyze these effects in depth the discussion section, where we examine how individual heads contribute to image generation, how their roles differ across layers, and how they interact when composed together.
Concepts be composed via head combinations. Interestingly, combining multiple heads for head-level guidance results the composition of their associated visual concepts. As shown in Fig. (b), the generated images exhibit hybrid effects-sure the combination of lighting (L1, H10) and shearing distortions oriented from top-left to bottom-right (L11, H15) suuggsstin that head-leve guidance serve mechanism for concept-level control. More examples of concept composition are provided Appendix E.
Composing heads increases image quality but may lead to over-saturation and over- simplification. Perturbing more heads strengthens the guidance effect but can also introduce undesirable artifacts such as oversaturation oversimplification As shown Fig. F (a), adding more perturbed heads results in over-perturbie wver-perturbation, leading to over-smoothed or overly simplified output. Fig. (b) further illustrates this effect quantitatively: the quality metric (FID (7 using 1K prompts from the MS COCO dataset begins to degenerate more heads are added after some threshold. This suggests that, in addition the choice of perturbation method, the number of perturbed heads itself a key factor for controlling the overall perturbation strength.
5

---

50
54
30
20
25
Top-k
(a) Samples guided by the top-k heads selected by HeadHunter.
(b) Qualitative results.
Figure 4: Results of Headeunter for general image quality improvement. Performance improves as more top-ranked heads are added, demonstrating the effectiveness of compositional head selection via HeadHunter. The dotted horizontal line in (b) indicates the best score achieved by layer-level guidance, which is surpassed by compact set of top-k heads for l < 10. Dashed lines indicate the FID of the best-performing layer-level perturbation for each guidance scale in Eq.
Controlling Head-Level Attention Perturbation
4.1 HeadHunter: An Iterative Framework for Retrieving Effective Attention Heads
The analysis in Sec. 3.11 revealed that individual attention heads exert distinct and often complementary influences via head-level guidance-sone some modulate lighting, others control color geometry. Moreover, these effects can be composed, meaning that combinations heads can yield richer and more controllable outputs than any single head alone. This raises key question: Can we guide image generation toward a user-defined objective by selectively perturbing attention heads that are automatically identified based on their individual and compositional effects?
To address this, we introduce Hunter framework that optimizes an arbitrary objective function by iteratively selecting a subset of attention heads to perturb. Each round consists of three stages: generation, evaluation, and expansion. In the generation stage, the framework perturbs attention maps for each candidate head (1, h) E S and generates samples using attention perturbation guidance with multiple prompt-seed pairs @ = ((1,, 81), (PM, 8ar)]. During the evaluation stage, computes the average objective score using the user-given objective function O over the generated samples for each candidate. In the expansion stage, top-k performing heads are added to the selected head set Sinal- This process is repeated for fixed number of rounds R. The full procedure is described in Algorithm|] [ We validate HeadHunter both quantitatively and qualitatively across tasks such as general quality and style enhancement.
This iterative design greedily selects attention heads in the order that most increases the objective. As result, it achieves rapid improvements during the early rounds (Fig. ( Later rounds enable the selection of heads that strengthen specific styles, such as imposing a particular tone on the image (Fig. (), even if they do not improve overall quality. We present detailed experimental results and analysis in later sections.
4.1.1 Improving general image quality
In Sec. we observe that even layers considered ineffective under layer-level guidance can improve quality when selectively activating appropriate heads. This suggests that fine-grained head-level selection across the entire set of heads may yield stronger enhancements in image quality. To explore this, we apply HeadHunter to search the head selection space.
Experiments. Various metrics can be used assess image quality, including vision-language models and learned reward functions. Following recent trends in diffusion models that optimize
image preference scores (49 27 62 65), we demonstrate our method using PickScore [27 as the
objective O. HeadHunter is applied with 20 prompt-seed pairs Q one round (R = 1), and selection of k = 24 heads per round. Additional implementation details are provided Appendix [] Qualitative results in Fig. (a) show that image quality progressively improves as more HeadHunter selected heads are included the guidance process, supporting the compositional nature head-level perturbations observed earlier. Fig. (b) presents quantitative results in terms of FID, evaluated on

---

Base line
r=
sunlit, glow, golden hour
showar profuseed
Baseline
cinematic lighting, dromatic tone
spring bloom, fresh vibrant colors
popcholllic prppy, colortil
Figure 5: Qualitative results of HcadHunter for style-oriented quality improvement. The first row presents the unguided result. In each subsequent row, we apply head-level guidance using additional attention heads that are incrementally selected by the HeadHunter framework. As more heads accumulated over rounds, the generated images exhibit progressively stronger alignment with the target style while improving visual quality. Original figures and additional results conducted in FLUX. 1-Dev 29 can be found in Appendix D.2
IK prompts from the MS COCO [ validation set. The performance improves as top-ranked heads are incrementally added, except in the high guidance scale case (w 6.0), where oversaturation may occur. Remarkably, using only 25% of the heads (k out of 24) achieves comparable to or even better performance than full layer-level perturbation (shown dotted lines for each guidance scale) and can be boosted much further by using additional heads. These results suggest that HeadHunter identify compact yet effective subset of attention heads, outperforming heuristic layer- based selection strategies.
4.1.2 Improving style-oriented quality
While Section 4.1.1 demonstrated HeadHunter's 's effectiveness in improving overall fidelity of samples,
many real-world applications require more targeted control over specific visual styles, such as evoking particular mood, or mimicking classical art techniques. Inspired by the findings of Sec. [3. and Fig. which reveal that certain attention heads are responsible for distinct stylistic or geometric attributes, we investigate whether HeadHunter can selectively enhance a target style through head- level guidance-wiin wwhle preserving or even enhancing overall image quality. We refer to this approach style-oriented quality improvement.
7

---

Experiments. To evaluate Headeunter's 's ability to enhance specific styles, we prepare two prompt sets: one for style (e.g.. "warm golden hour glow") and one for content (e.g.. "portrait of a violinist") Each trial uses composite prompt of the form "style, content." We run HeadHunter with R = 5 and h: = 3. Fig. shswws that, as more heads are selected, the generated images increasingly exhibit the intended style while maintaining structural integrity.
Fig.
reports quantitative metries: PickScore 27 and the LAION Aesthetic Score (AES) 49 both improve steadily as k
21.50
increases in the prompt-seed pairs Q. These
23.25
results confirm that HeadHunter effectively amplifies target styles in aligmment with human preferences. For additional qualitative examples with SD3 [4] and FLUX.1-Dee [29], we refer readers to Figs.
PickScare
23.00
6.0
20.75
AES
5.9
20.50
20.25
5.8
29
Appendix.
5.7
Round
Round
Assessing generalizability to unseen content prompts. To evaluate the generalizability of HeadHunter, we compare it against both the baseline and CFG using a set of 50 unseen con- tent prompts. As shown in Tab. HeadHunter achieves significantly higher human preference scores than the baseline and performs comparably to CFG, thereby validating its ability to generalize to novel content prompts. Notably, as illustrated Fig. 7 HeadHunter leads to a substantial enhancement of stylistic attributes even compared to CFG. For the style prompt sunlit, warm glow and golden hour, HeadHunter produces visibly intensified reddish tones and sunlight effects. Likewise, for the line drawing ... style prompt, monotone line characteristics are markedly enhanced. These results suggest that HeadHunter can serve as an effective plug-ane and---dd-ppp module within existing inference pipelines (including those employing CFG) to improve both stylistic fidelity and overall image quality without requiring additional training.
Figure 6: Quantitative results of HcadHuuter for style-oriented quality improvement. As more heads accumulate, the generated images progres- sively align better with the target style and exhibit improved visual quality.
4.1.3 Discussion
Surprising utility of individually weak heads. Interestingly, some heads that individually produce poor outputs can still play an essential role in composition. In Fig. S (a), we observe a sudden enhancement in stylistic expression (e.g., intensified red) with inclusion of certain head set. By individually performing head-level guidance with each head in the set, we found that some head alone yields low-quality outputs but it reliably enforces style trait. Such heads unlikely to be selected through one-shot evaluation due to low quality, but HeadHunter's iterative nature allows them to be integrated later their utility emerges in composition. This highlights HeadHunter's ss capacity to exploit compositional synergies among heads.
4.2 Controlling Guidance Intensity via Attention Map Interpolation
Although our head-level guidance provides fine fine-graine grained, objective-specife control, we propose an orthogonal approach that enables continuous modulation of the guidance effect. This allows for more precise adjustment of the perturbation intensity applied to selected heads or layers. Specifically. we introduce a simple interpolation strategy that interpolates original attention map with its perturbed counterpart. For example, the interpolated attention map for perturbed-atrentio guidance (PAG) 1 is defined as: (SonPAG)
u) WI, for (1, E S, u [0, 1].
This formulation replaces A (PAG) in Eq. with its interpolated version, enabling a smooth transition 1,i between the original attention map and the fully perturbed one.
We present qualitative results across varying u values in Fig. 9 These results show that increasing the interpolation parameter u initially enhances sample quality by rectifying structural artifacts in the samples. However, beyond certain point, it leads to oversimplified images with smoothed backgrounds and exaggerated structures. The interpolation method provides "sweet spot" where the samples exhibit improved structure while retaining visually realistic features. By treating the

---

Table 1: Quantitative evaluation of generalizability to content prompts. Number in the parenthesis denotes guidance scale w. Applying HeadHunter (style-oriented quality setting) to unseen content prompts demonstrates strong generalization, yielding significantly higher human preference scores than the baseline and performance comparable to CFG.
Method
PickScore
AES
HPS
Imreward
Baseline
19.66
5.37
0.2147
-0.591
CFG (3.0)
20.87
5.71
0.2924
0.844
CFG (6.0)
20.92
5.80
0.3046
1.063
HeadHunter (3.0)
20.70
5,92
0.2901
0,470
CFG (3.0) HeadHunter (3.0)
20.92
5.93
0.3036
0.845
Poouline
CP06.0
Headiummer 3.0
Headitunee 3.0
CRC
cincmatic lighting, Sone
Janiit glou, goldon
vireree sovers
coledal
endlen wandenlly proplic mortor gregllec
Figure
Qualitative evaluation of generalizability to unseen content prompts. Applying HeadHunter (stylc-oriented quality setting) results in substantial enhancement of stylistic attributes, outperforming not only the bascline but also CFG.
(LO H17)
(L18 H11, LO H17,
(L18 H11, LO H17, L2 H3)
(a) "Sudden style change" moment and participating heads at that moment
(b) Detailed analysis how cach attention head individually contributes to guidance
Figure 8: Role of individually weak heads. For sudden stylistic transition moments (e.g., adding global warmth in (a)). we visualize the effect of newly added heads in (b). One of these heads generate blurry reddish outputs when used alone (see red box). Although they fail to produce meaningful content when used independently, they contribute effectively when composed with previously selected structural heads. This highlights the importance iterative strategy since those heads are unlikely to be selected by one-shot evaluation due to low quality. Additional results can be found in Fig. 21

---

==0 (A)
u= 0.2
0,4
u 0.6
u 0.8
== 1.0 1I
"Old South American man smiling in golden hour"
"teenage girl with auburn hair sketching beside a foumtain"
Figure 9: Generated Images with linear interpolation between attention map A and identity matrix I. While increasing enhances quality up to a point, eventually results in over-saturation and structural over-simplification. Detailed hyperparameters can be found in Appendix G
adentitt-matrii perturbation in PAG [ as a specific point in probability distribution space, our formulation enables a smooth and principled interpolation between A and I. The scalar u E [o. 1] provides an intuitive and interpretable control over perturbation strength. Additionally, since this method is a simple linear interpolation of self-attention maps it can be implemented with a few lines of code. We refer to this controllable variant of PAG as Soft Perturbed-Attention Guidance (SoftPAG).
Interpolation parameter vs. guidance scale w. Both the guidance scale w and interpolation parameter u influence the strength of perturbation, often affecting saturation and structural fidelity. This raises the question of whether adjusting u offers any advantage over simply tuning w.
Fig. 10 in Appendix visualizes metric values over a grid of (w, u) pairs, where brighter regions indicate better performance and red boxes denote optimal configurations. Across most metries, the rest-performing setting occurs at < 1.0, indicating that full replacement (i.e., v 1 as in PAG) often suboptimal. These results underscore the importance of explicitly controlling perturbation strength via interpolation, rather than relying solely on the guidance scale. Due to space limitations, we defer a detailed discussion of the effectiveness of SoftPAG and the selection of and to Appendix B
Conclusion
In this work, we move beyond heuristic layer selection in attention perturbation guidance by identify- ing attention heads as more meaningful and fine-grained unit of intervention. To the best of our knowledge, this is the first work to perform attention perturbation at the level of individual heads. Specifically, we present HcadHunter, an iterative framework for selecting semantically relevant attention heads aligned with arbitrary, user-defined objectives. Empirical results on state-of-the- DiT-based models, including Stable Diffusion 3 and FLUX.1, show that head-level perturbation not only improves general image quality with single search but also aesthetically enhances spe- cific visual styles. In addition, we introduce SoftPAG, lightweight yet powerful mechanism for continuously modulating perturbation strength via attention map interpolation. Together, these two approaches address key limitations of prior guidance methods, enabling more targeted, effective, and controllable inference-time interventions. We believe this work opens promising directions for interpretable and modular control in generative modeling.
10

---

Acknowledgments and Disclosure of Funding
This research was supported by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-11190075. RS-2024- 00509279, RS-2025-I121 22068, RS-2023-00227592. RS-2025022 14479, RS-2024-00057882, RS- 2025-2544 1838, RS-2025-25444 2125-25441838, RS-2025-0221 14299, RS-2025-02217255) and the Culture,
Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism IRS-2024-00345023
RS-2023- 00222280, 20S20033000,65009, and National Research Foundation of Korea RBS-2024-00346557))
References
[1] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In European Conference on Computer Vision, pages 1-17, Springer, 2024.
[2] Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemehinov, Kfir Aberman, Dani Lischinski, and Daniel Cohen-Or. Stable flow: Vital layers for training-free ffree image editing. arXiv preprint arXiv:2411. 14430, 2024.
[3] A. Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, and Dominik Lorenz. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, abs/2311.15127, 2023.
[4] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22560-22570, 2023.
[5] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74-91. Springer, 2024.
[6] Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-0: Fast and controllable image generation with latent consistency models. arXiv preprint 200V::::0010055522 2024.
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint 10.00223 2023.
[8] Hyungjin Chung, Jeongsol Kim, Michael T. McCann, M. Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. ArXiv, abs/2209. 2022.
[9] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and J. C. Ye. Improving diffusion models for inverse problems using manifold constraints. Neural Information Processing Systems, abs/2206.00941, 2022.
[10] Mete Demircigil. Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-228 299, 2017.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural information processing systems, 34:8780-8787 8794, 2021.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint 2010.1000011922 2020.

---

[13] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly. et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
[14] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[15] fal. Auraflow: flow-based text-to-image generation model. https://huggingface. co/ Fal/AuraFlon 2024, Accessed: 2025-05-14.
[16] Yossi Gandelsman, Alexci A Efros, and Jacob Steinhardt, Interpreting clip's image representa- tion via text-based decomposition. arXiv preprint arXiv:2310009916, 2023.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. arXiv (Cornell University), 2017.
[18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, abs/2210.02303, 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 1.Xiv:2207.1259,, 2022.
arXiv preprint
[21] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.33458, 2022.
[22] Susung Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. Advances in Neural Information Processing Systems, 2024.
[23] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7462-7471, 2023.
[24] Junha Hyung, Kinam Kim, Susung Hong, Min-Jung Kim, and Jaegul Choo. Spatiotemporal skip guidance for enhanced video diffusion sampling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 11006-11015, 2025.
[25] Zahra Kadkhodaie and Eero P. Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Neural Information Processing Systems, 2021.
[26] Tero Karras, Miik Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. Advances in Neural Information Processing Systems, 2024.
[27] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to- to-imaee generation. Advances in Neural Information Processing Systems, 36:36652-36663 2023.
[28] T. Kynkäänniemi. M. Aittala, Tero Karras, S. Laine, Timo Aila, and J. Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. Advances Neural Information Processing Systems, abs/2404.07724, 2024.
[29] Black Forest Labs. Flux.1 [dev]: A 12b parameter rectified flow transformer for text-to- image generation. httpp://hnggingffc...oollleeeeee Accessed: 2025-05-14.
2024.
12

---

[30] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Arsh Koneru, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Reflect-dit: Inference-time scaling for text-to-image diffusion transformers via in-context reflection. arXiv preprint arXiv:2503. 12271, 2025.
[31] Tiancheng Li, Weijian Luo, Zhiyang Chen, Liyuan Ma. and Guo-Jun Qi. Self- goffaneid Boosting flow and diffusion generation on their own, arXiv preprint arXiv:4412.0887, 2024.
[32] Chen-Hsuar Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Y Yu Liu, and Tsung-Y Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 300-309, 2023.
[33] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[34] Y. Lipman, Ricky T. Q. Chen, Heli Ben Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations, abs/2210.027477 2022.
[35] Yaron Lipman, Ricky TQ Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02774,, 2022.
[36] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2022.
[37] Nanye Ma, Larry B. Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. European Conference on Computer Vision, 2024.
[38] Nanye Ma. Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.097.22 2025.
[39] Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. Dreammatcher: appearance matching self-attention for momantically-consistie text-to-image personalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8100-8110, 2024.
[40] Jungwon Park, Jungmin Ko, Dongnam Byun, Jangwon Suh, and Wonjong Rhee. Cross-attention head position patterns can align with human visual concepts in text-to-image generative models. arXiv preprint arXiv:2412.02223 2024.
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[42] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv (Cornell University), 2023.
[43] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 2022.
[44] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprini arXivv2008..222 2020.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
13

---

[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computiig and mmpputer-asiisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 231-241. Springer, 2015.
[47] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradely, Otmar Hilliges, and Romann M. Weber. Cads: Unleashing the diversity of diffusion models through condition-anneale sampling. International Conference on Learning Representations, 2024.
[48] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. S. Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to- image difffusion models with deep language understanding. Advances in Neural Information Processing Systems, abs/2205. 1487, 2022.
[49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25294, 2022.
[50] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308, 16512, 2023.
[51] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurüi Makarov, Filippos Kokkinos, Naman Goyal, A. Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. Text-to-4d dynamic scene generation. In International Conference on Machine Learning. 2023.
[52] Jascha Dink-Dick,tein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. pmllr 2015.
[53] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv (Cornell University), 2015.
[54] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv, abs/2010.02502, 2020.
[55] Jiaming Song, Arash Vahdat, M. Mardani, and J. Kautz. Pseudoinverse-guided diffusion models for inverse problems. International Conference on Learning Representutions, 2023.
[56] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Neural Information Processing Systems, pages 1 11895-11907, 2019.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13466, 2020.
[58] Yang Song, Jascha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv (Cornell University), 2020.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[60] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, -4406 2023.
[61] Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T. Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-vies video diffusion models. arXiv preprint 3 2025. 2025.
14

---

[62] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 19903-1555,3, 2023.
[63] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang. Jiazheng Xu, Yuanming Yang. Wenyi Hong. Xiaohan Zhang. Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang. Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. International Conference on Learning Representations. 2025.
[64] Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, László A. Jeni, S. Tulyakov, and Hsin-Ying Lee. 4real: Towards photorealistic 4d scene generation via video diffusion models. Advances in Neural Information Processing Systems, abs/2406.07472,, 2024.
[65] Sixian Zhang. Bohan Wang, Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimensional human preference for text-to-image generation. Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8018-8027, 2024.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: substantiate our claims in Sections with results included in the section.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in the separate section of Appendix.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yss]
We provide all the experimental details of our work. Our work relies on open models and public codebases. Since we don't conduct any training, our results are not tied to any specific datasets.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will make our code publicly available along with instructions.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings, including network architecture, guidance scale, and sampling step are detailed Section G
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Although we conducted experiments using multiple evaluation metrics, we did not report statistical significance
error bars.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
16

---

Justification: We include this information in the Appendix.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pwp.aaaaapc.ccoaaaaaaaaaa
Answer: [Yes]
Justification: The research involves algorithmic development and evaluation standard optimization benchmarks. does not involve human subjects or obviously ethically sensitive applications, and assume it conforms to the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper focuses on technical contributions and does not include a specific discussion of broader positive or negative social impacts.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work is analysis-driven. We leverage open models and open codebases to guide and craft our experiments.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We make sure to credit all the sources of the assets we used in our work.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper do not provide new assets.
Guidelines:
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects, therefore IRB approval is not applicable.
16. Declaration of LLM usage
17

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We used LLM only for writing, editing and formatting purposes.
18

---

Appendix
This appendix provides supplementary material support the main paper.
Contents
A Related Works
1
B Continuous control on perturbation strength via attention map interpolation
2
Other perturbation methods our framework comparison
Uniform Guidance)
4
Smoothed Energy Guidance
5
Softmax temperature scaling
6
D
Headeunter: Implementation and additional analysis D.1
11
Implementation details]
11
Additional results analysis
E
Additional results and analysis of head-level perturbation guidance
22
E.1
Additional results of individual head level guidance
23
F.2
Additional results on interpretable head-level guidance
25
E.3
Additional results on the combinational effect of head level guidance
27
F
Head-level perturbation guidance with different perturbation methods
27
Experimental details
30
H Experimental resources
30
Limitations & Broader Impacts
30
A Related Works
42 have become the foundation and video 18 63). Early methods rely on stochastic differential equations (SDEs) learn reverse denoising process from noise to data. More recently, deterministic samplers based on ordinary differential equations (ODEs), including rectified flow [37] [4] and flow matching [34], have emerged as efficient alternatives, learning continuous trajectories that transform noise into data. These methods accelerate convergence and improve stability, especially in large-scale models. In parallel, network architectures have shifted from U-Net backbones [42 to Diffusion Transformers (DiT) 41 14 7), enhancing scalability and representational capacity.
Diffusion models. Diffusion models 19 54
56
of modern generative modeling, driving advances in both image 45
synthesis [2]
Attention perturbation guidance. In recent studies, various modifications and extensions of classifier-free guidance have been introduced. Among them, methods that directly perturb the attention layers for guidance work effectively. Self-Attention Guidance (SAG) (23) applies Gaussian blur to the self-attention layers of the UNet to introduce perturbations. Similarly, Perturbed-Attention Guidance (PAG) [I] generates perturbations by replacing the attention maps with an identity matrix. Spatiotemporal Skip Guidance (STG) 24 is a 3D extension of PAG that improves sample quality by selectively skipping spatiotemporal layers. Smoothed Energy Guidance (SEG) (2 defines the energy of self-attention and employs 2D Gaussian blur to reduce the curvature of the energy landscape, using the result as form of perturbation. Autoguidance (26) enhances image quality by using bad version of the same conditional model as source of perturbation. Self-Guidance (SG) 31 leverages the model prediction at noisy timestep to generate perturbations.

---

B Continuous control on perturbation strength via attention map interpolation
Layer
Layer
Layer
Layer
Layer
Layer
1.0
1,0
0.8
0.9
15.0
0.4
0,4
0.2
1.0
0.8
*M
0.3
0.4
0.4
0.2
18
11.0
0,
02
10
18
04
2.0
0.5
2.0
4.0
8.0
2.0
4.0
6.0
800
20
4.5
20
20
Figure 10: Grid search for SoftPAG, on guidance scale w and interpolation parameter with different metries.
d=0.25
u=0.50
56
u=0.75
52
Global Min RID (=0000 k=
Top-k
Figure 11: Effect interpolation parameter u on FID across PickScore top- head selections. As u decreases, the perturbation becomes milder, shifting the optimal point (lowest FID) toward larger & (i.e., more heads needed). Notably, 0.5 yields the best FID (indicated by the red dashed line), highlighting the benefit of moderate perturbation strength.
Pre-selecting interpolation parameter u in HeadHunter search. As running HeadHunter multiple times with different interpolation parameters is computationally expensive, is generally sufficient to search for heads using the original PAG. Nonetheless, we can explore the effect of u through HeadHunter with the same setup as in Sec. 1.1.1 replacing PAG with SoftPAG (Eq. 7 and varying u. The results, shown in scores as function of for different values.
single-round search to study general quality trends. To this end,
Fig. II report FID
We observe that lower u values (i.e., softer perturbations) require more heads to achieve optimal performance, but also tend to yield better overall results with moderate setting, Empirically, we find that u = 0.5 and
offer a strong trade-off.
This supports the view that both and k modulate the strength of guidance. While this paper introduces two axes of fine-grained
selection and attention map interpola- tion-tuning is significantly more efficient than rerunning head selection. In practice post-selecting after standard head search is typically sufficient. However, if more compute is available, one may rerun HcadHunter with moderate u (e.g., 0.5) for refined selection.

---

0=5.3 u= ==
minlit warm glow, goiden hour
conemttic lighting dramatic lone
Figure 12: Incorporating SoftPAG with the final heads retrieved by HcadHunter for style- oriented quality improvement. Setting the interpolation parameter = 1.0 makes the effect of head-level guidance more visible, which helps in selecting effective heads via HcadHunter. However, it may also introduce unwanted artifacts such as oversaturation or oversimplification Post-tuning after head selection effectively reduces these artifacts while preserving the style enhancement, with minimal additional cost.
Post-sclecting of for artifact removal with retrieved heads. As discussed in Sec. 3.1 the number of heads used head-level guidance affects how strong the perturbation is. When more heads are used, the intended effect such as style enhancement can be amplified. However, this may also lead to unwanted side effeets including oversaturation or oversimplification due bias of reward models (27 49). In the HcadHunter framework, heads are added step step. This helps reinforce the target objective, but may also increase the risk of artifacts. To better control the strength of guidance, SoftPAG, which allows
mitigate the artifacts through the interpolation parameter
As tuning both the number of heads and the interpolation parameter at the same time can be complex and computationally expensive, we adopt simple two-stage approach. First, we use HeadHunter with u = 1.0 to select effective heads. A larger helps make each head's effect more visible, which helpful during selection. Then, in the second stage, optionally reduce to fine-tune the strength of the guidance and reduce any unwanted artifacts.
This approach offers practical trade-off by eliminating complex tuning while retaining control over the final output. In Fig. [2] we show examples where retrieved heads with u 1.0 introduce unnatural artifacts. Reducing the SoftPAG parameter to u = 0.5 effectively removes these artifacts while preserving the style enhancement provided by the selected heads.
3

---

C Other perturbation methods in our framework and comparison
In the main paper, we focus on a single representative perturbation method: identity matrix replace- ment (PAG). Within our framework, which interprets the attention map as probability distribution, identity perturbation can seen as one end of the autropy-modulatinn transform that minimally shifts the distribution's s entropy. Fig. 13 provides conceptual overview of this unified view, with other perturbations. Our key insight is twofold: (1) attention perturbations can be understood as transformations over probability distributions, and (2) the sfrength of any perturbation can be smoothly controlled via interpolation with the original attention map. This enables elegant integration of existing methods while offering controllability.
X
Xc-1
V/
fa(x))
Ba
Cooditio
Full-Attemtion
Es Coe)
?
Sual
K
x1
Perturbed Attention
(a) Classifict-Free Guidance
(b) Unified Attention Perturbation Guidance
entrapy
High entropy
(1 U)A
UU u)M
Uniffrmm
uA u)A
(SEC + (1 u)A
A original
Al
A SEG
Max Guidance
attention map
<Attention Perturbation Space>
(c) Perturbation methods
Figure 13: Unified attention perturbation guidance. Our framework unifies a variety of perturbation strategies by interpreting attention maps as probability distributions and interpolating between the original and perturbed variants.
C.1 Uniform Guidance
From an information- ceammation-thhoretiic perspective, the identity matrix attention map a special case where each row of the attention map corresponds to Dirac delta distribution, which is the lowest possible entropy case, the opposite extreme the entropy spectrum lies the uniform distribution U, whose rows are all ...... I ; it attains the maximum entropy, allowing each query to attend equally to every key. Replacing interpolation with the identity matrix in SoftPAG,
by in selected heads gives Uniform Guidance (UG). Similar to the define distributional perturbation between the original attention matrix and via linear interpolation:
A (SeRUG) = (1 u) uU, u [0.
(8)
We show the interpolation results Fig. Interestingly, this perturbation strategy tends to preserve the original structure of objects in the unguided sample (u 0) slightly better. However, it often over-restores artifacts present in the original structure, sometimes generating unintended elements (c.g., gg,&& framed portrait in the background of Row 1, swimming pool in Row 3). Both methods,

---

(4) 044
0.8 1 (U)
Figure 14: Generated Images with linear interpolation between attention map A and a uniform matrix U. While increasing enhances quality up point, it eventually results in over-saturation and structural over-sinplification Detailed hyperparameters can be found in Appendix G
however, may exhibit oversaturation or oversimplification when overly strong perturbations are applied, highlighting the importance of balanced interpolation with the original attention map A.
C.2 Smoothed Energy Guidance
Smoothed Energy Guidance (SEG) [22] applies 2D Gaussian blur along the query axis to smooth the attention logits. By adjusting the kernel width a, users can naturally control the strength of the perturbation. In the extreme case where a - xo, SEG reduces to averaging all query features and applying the result globally.
SEG is not directly applicable to MMDIT. However, SEG is not directly compatible with MMDiT [14, where both image and text tokens participate in attention. Applying a 2D Gaus- sian blur over the query axis becomes nontrivial when attention spans across modalities, in recent text-to-video models where multiple frames attend jointly. This issue becomes more pronounced with the rise of multimodal attention -MMattention).
Restoring controllability to SEG using our framework. To address this, we propose simple yet widely applicable solution: interpolate between the original attention map A and the one computed from the mean query
to the limiting case of SEG with
Specifically, we interpolate between the original attention map A and the SEG attention map A (SEG)
computed using query vector, follows:
(SoftEE)) = (1 A + UA (SEG)
u & [0 [0,
(9)
where
QK
N Σ
A (SBG)
Softmax
Q = Ivq Nxd
q
Q ERd.
(10)
Vd
Here, Q Nxd is the query matrix, and Q Nxd replicates the mean query vector across all N tokens. This allows for a smooth transition from the token-specific attention map to globally- averaged variant, effectively reintroducing controllability.
This provides controllable and modality-agnostic extension of SEG to architectures like MMDiT, where traditional Gaussian blurring becomes infeasible. We show the interpolation results in Fig. 15

---

(A)
0.8
(A (SBG)
Figure 15: Generated images with linear interpolation between attention map A and smoothed attention map A (SEG)
C.3 Softmax temperature scaling
Attention perturbation using softmax temperature scaling. One intuitive way to increase or decrease entropy is by introducing temperature parameter 7 into the Softmax operation of attention,
as follows:
0.01
0.2
0.5
T= (A)
==2
T=
100 (U)
Figure 16: Generated images with temperature scaling perturbation.
log A T
A (temp) softmax
0.
(11)
Lower temperatures (r - 0) sharpen the distributions, driving them toward the Dirae delta distribu- tions, while higher temperatures (r oo) flatten the distribution towards uniform. This formulation provides a unified mechanism to traverse both entropy-increasee and entropy-decreasing directions within the attention perturbation space.

---

(A)
0.8
u (A(T0))
Figure 17: Generated images with linear interpolation between attention map A and smoothed attention map A (T)))
Perturbation - 0 results in diff'erent perturbation with PAG (I). As - oC, the distributions approach uniform distributions, following an exponential-geet -cciiesi path. In contrast, as distribution converges to a Dirac delta distribution. However, unlike the identity matrix I, where all probability mass is placed each query's 's own index, this now-temperature limit concentrates mass on the maximum entry of each row, potentially differing from the diagonal. We illustrate the effect of varying in Fig. [6 Notably, increasing and decreasing 7 leads meaningful quality improvements. suggesting that modulation of entropy in either direction can enhance generation.
the
Intuitive control parameter using interpolation framework. Softmax temperature scaling pro- vides a simple way modulate perturbation strength by adjusting the temperature E (0, oo). However, it introduces practically unintuitive control parameter and the limiting distribution as т - 0 theoretically unreachable in practice due to numerical instability.
Our interpolation framework offers a compelling alternative: it allows for control using a normalized and intuitive parameter [0, 1], while enabling us directly realize the theoretical limit of the attention map as T - 0, denoted as A T-0 without the numerical issues associated with temperature scaling.
Concretely, in the limit r + O, the softmax attention converges to a one-hot distribution where, for each row, the position of the maximum logit becomes 1 and all others become 0:
A (T-))
{1
if arg max Ask
(12)
otherwise.
Then the perturbation methods is as follows:
A (SoRMG) (1 u)A + UA (T-))) u u [0.1)
(13)
As shown Fig. 17 our framework can replicate this limiting behavior while preserving control- lability through continuous interpolation parameter. Compared to softmax temperature scaling (Fig. [16). this approach offers more intuitive and stable mechanism for navigating the space of attention perturbations. We refer to this perturbation- guidance as Max Guidance (MG).
Quantitative comparisons. We discuss various perturbation strategies and present their qualitative results in Fig.
To complement these findings, we provide quantitative comparisons
in Tab. 2

---

laye?
Loper
Loyer
Layer
Layer s
Layer
Laver
1.9
1,
0.8
0.8
0.4
04
353
02
1.
1,
08
55
04
0,
02
19
110
02
04
02
318
1e 0,66
21
02
2.0
4.0
85
2.0
40
20
46.60
60
20.46 60.80
W
Figure 18: Grid search for UG, guidance scale to and interpolation parameter with different metrics.
Note that each perturbation method may benefit from different optimal hyperparameter configurations. To ensure a fair evaluation, we define parameter pool and report the best-performing setting for each method and each metric. Specifically, namely layer 7, layer 8, layer 9, and layer 10; guidance scales w E (2.0, 4.0, 6.0, 8.0); and interpolation parameters
consider perturbation layers the early-to-mid range.
{0.2, 0.4, 0.6, 0.8, 1.0].
As shown in the Tab. 2] the I perturbattion consistently yields strong performance across most settings. Based on this robustness, adopt it primary analysis tool in the main paper (Sec. 3
Method
FID 1
Precision 1
Recall
Density
Coverage f
PickScore
ImageReward t
MPS
Uniform Matrix U
53.37
0.62
0.55
0.89
0.05
21.90
0.53
11.21
Query Mean A (SGG)
53.33
0.62
0.53
0.91
0.06
21.89
0.50
11.13
Identity
Table 2: Comparison of perturbation strategies across different metrics. Each method shows strengths in different metrics. Overall, Identity matrix perturbation (SoftPAG) achieves strong results across most quality, diversityand human preference metrics. Notably, identity matrix perturbation outperforms others in preference- scores such as PickScore, indicating better alignment with human perception and more visually pleasing results. Note that FID does not always correlate with human judgment.
More guantitative results. We measured the quality of images generated by layers 7, 8, 9, and 10, which produce the highest-quality outputs. Fig. [19 depicts the results of SoftPAG, which interpolates between the identity matrix and the attention maps, while Fig. depicts the results of UG, which interpolates between the uniform matrix and the attention maps.

---

wed
Layer 7
54
p-4t
240
2015
labn wathell
A32
13.00
148
Layer 8
53
RecCooe
mageRoward
MP5
Dendly
Coverage
6-10
040
predsion
Non_zesthedic
340
345
Layer
3100
28
14
64
43
10
42
12
Moddore
mageRoworr
Deroit
Coverage
04
14
14
06
52
Layer 10
12
2431
33
Roldoo
mapehaward
MPS
0/6
0.5
6950
Figure 19; Quantitative results of interpolation in SoftPAG, on layer 7,8,9, 10 with different metrics.

---

2007
E415
Layer 7
3233
PetScere
Imagefionard
MP2
-22
wx80
42 torchmetreee
prec
recal
MCO seathete
dip crom
25.00
21.3
Layer 8
F:15
M40
PrcSccre
InageRoward
MP5
Dundiy
Gaverage
A020
1.00
04
03
04
12
08
08
12
23
64
68
6354
33
Layer
InapcRoward
0.31
1.4
p-15
v-4
031
24
swathalic
36
374
253
Layer 10
32
352
352
245
34
50
PiaSowe
Irog Reaord
MPS
Eenddhy
Coverage
0.51
010
Figure 20: Quantitative results of interpolation in UG, on layer 7,8,9. 9, 10 with different metrics.
10

---

D HeadHunter: Implementation and additional analysis
D.I Implementation details
Algorithm HeadHunter: Iterative Objective-Aware Head Selection
Require: Diffusion model M, objective function O, prompt-seed pairs Q
((11, ...; (PM-&8))) attention head set (44,)),..... (L,, hH,),, number of heads per round k, number of rounds R Ensure: Selected heads Stial
1: Stinal +- 0
> Initialize selected head set b Initialize remaining head pool
2: + SS
3: for = to R do
4:
C+
Initialize temporary score list
5:
for each (1, E R do Generation
6:
7:
Surget < Sinal U [(t,))
Define perturbation set
8:
Generate { MM using M with perturbation guidance Starget and (pj, 8j) E Q
9: Evaluation
10:
8(1.) + M -== (b),2;)
D. Compute average objective score
11:
Append ((1, h), ((,))) to
12:
end for
13:
Expansion
14:
Sort C by S(,A) in descending order
Snew t top-k heads from C
16:
Súnal + Stial U Snew
R R Snow
18:
end for
return Siinl
Improving general image quality. To run HeadHunter, we used 20 prompts (M = 20), with 24 and R 1. All prompts and the corresponding seeds are listed in Tab. Since SD3 contains 24 layers with 24 attention heads per layer, we consider the full set of N = 24 x 24 = 576 attention heads. For head-level guidance, we set the guidance scale 3.0 and the number of inference steps to 20. Additionally, we set the interpolation parameter 0.25 in Eq. 7 to avoid excessive over-smoo effects.
While it is possible to set R > 1 to run HeadHunter iteratively, we found that single round suffices in practice. This is based the compositional nature of head-level guidance. Heads that individually improve quality tend to combine well, yielding consistent improvements in overall generation quality. As shown in Fig. even a compact set of heads selected in single round can outperform the best-performing layer-level guidance.
Improving style-oriented quality. For SD3, we evaluated total of 55 style prompts for both quantitative and qualitative analysis. Since SD3 consists of 24 layers with 24 attention heads per layer, considered all N 24 x 24 576 attention heads. For head-level guidance, we the guidance scale to 3.0 and used 20 inference steps.
used total of 23 style prompts. The model comprises 57 layers with 24 attention heads per layer, resulting in = 57 x 24 = 1368 attention heads. Head-level guidance was applied with guidance scale of 8.0 and 15 inference steps.
For FLUX.1-Dew.
In both cases, we used content prompts (M 5) per style. The full list of content prompts provided in Tab.

---

D.2
Additional results and analysis
D.2.1
Additional results
We present additional qualitative results for style-oriented quality improvement using HeadHunter in Figs. 24 to 26 Each row corresponds to the result obtained by perturbing the head set selected at round r of Alg The top row shows unguided generation, and the lower rows show results with an increasing number of selected heads applied. As more heads are progressively added, the generated images align better with the target style while maintaining visual plausibility.
For example, for the style prompt "sunlif, warm glow, golden hour", we observe an increasing glow effect, culminating in a global golden- houlnn-hour& tone from the fourth row onward. We provide a more detailed analysis of the contributing heads in Sec. 4.1.3 and Fig. 21
Interestingly, the framework effective not only for photorealistic styles such as "cinematic lighting", but also for abstract or simplified styles like "line art" or "flat paper cur". Since this style adaptation achieved without modifying any model parameters, enables a reusable and lightweight preference tuning.
HeadHunter can benefit from with model size. Since larger models have more expressive attention heads, they can be more effectively leveraged in head-level perturbation guidance. In Fig. 24 we provide qualitative results of style-oriented quality improvement conducted on FLUX. dedev 29
D.2.2 Additional analysis
Distribution of selected heads across layers. We investigates the layer distribution of selected attention heads within a model's architecture, specifically examining whether impactful heads are localized particular layers and if their distribution varies by objective. For the general quality setting, we select the top-15 heads per prompt and aggregate them, excluding duplicates (Fig. 22 (a)). For style-oriented quality setting, we repeat the process per style (Fig. (b)).
Our findings reveal that effective heads are not concentrated in any single layer. Even the most frequently selected layer accounts for less than 12% of all chosen heads across both objectives (Fig. 22 (a). (b)). This dispersed distribution suggests that a layer-wise guidance approach, which uniformly perturbs all heads within a layer, may be suboptimal. Such approach risks overlooking critical heads in other layers while simultaneously applying modifications to irrelevant heads.
Furthermore, we observed distinct distribution patterns dependent on the objective. Heads identified for improving general image quality predominantly reside in early to mid-layers, with a mere 2.9% located in layers 2 16. In contrast, heads contributing to style-oriented quality improvement exhibit a significantly broader distribution, with 24.8% found deeper layers (2 (216).
We further examine the evolution of head distribution during HeadHunter's iterative refinement for style-oriented quality setting in Fig. 22 (c)). Initially, selected heads are concentrated in mid- layers, mirroring the pattern observed for general quality setting. However, as rounds progress, the distribution become notably flatter, indicating broader engagement of heads across layers for stylistic refinement. This quantitative shift aligns with qualitative observations.Ar shown in Fig. 24 (spring bloom, fresh vibrant colors), Fig. 25 (cinematic lighting, dramatic tone), and Fig. 26 (line art drawing ...). the second row (Round 1) shows improved image quality but little stylistic traits. However, from round onward, the style becomes much more pronouneed..
In conclusion, our research indicates that (1) effective heads are not localized to specific layers, thereby challenging the efficacy of layer-level guidance, and (2) head distributions are objective- dependent, underscoring the necessity of objective-specific selection. These findings collectively emphasize the critical role of targeted head-retrieval and guidance frameworks, such as HeadHunter, in fully harnessing the diverse functionalities of attention mechanisms.
Does the style enhancement occur even without style prompts? In our style-oriented quality improvement experiments, we search for heads that consistently enhance a given style by using composite prompts of the form "style, content". Then a natural question arises: Do these heads still express their stylistic traits when the style is not explicitly specified in the prompt or even under unconditional generation?
12

---

CI
(L18 H11, Lo H17, L2 H3)
(LO, H2, L13 H19, L5 H12)
sunlit, warm glow, golden hour
spring bloom, fresh vibrant colors
(a) "Sudden style change" moment and participating heads at that moment
{L18 H11}
{LO H2}
{LO H17)
(L13 H19}
H3}
{LS H12]
(L18 H11, H17, H3}
{LO H2, H11, H3}
(b) Detailed analysis each attention head individually contributes guidance
Figure 21: Role of individually weak heads, For sudden stylistic transition moments (e.g.. adding global warmth pink hues shown in (a)), visualize the effect of newly added heads in (b). One of these heads generate blurry reddish or pinkish outputs when used alone (see red box). Although they fail to produce meaningful content when used independently, they contribute effectively when composed with previously selected structural heads. This highlights the importance of iterative strategy since those heads unlikely to be selected one-shot evaluation due low quality.
To explore this, we apply the heads selected by HeadHunter in an unconditional setting (i.e., with null prompt, ""). As shown in Fig. (a), we find that even with a null prompt, using these heads can still induce visual traits that align with the original style to some extent. This suggests that certain heads may encode intrinsic stylistic properties that influence generation regardless of prompt or initial noise.
Yet, this effect style traits are strongly expressed with style prompts, they become weak or ambiguous when only content prompts are used or under unconditional generation. This may be due to polysemanticity in head behavior. This emerging style consistency when using style prompts indirectly supports the validity of including style prompts during HeadHunter's search process in style-oriented quality setting.
guaranteed across all styles. As can be seen in Fig. 23 (b), in some cases, while
Computational efficiency. HcadHunter requires one-time search process to identify effective attention heads for a given objective. If we denote the number of prompt-seed pairs as M, total attention heads as N, number of rounds as R, and the costs for generation and evaluation as Ogen and Ceval, the overall cost is approximately M MN(CCen + Ceva) R. our experiments using stable-diffii --i-dfiffiio---- --mddimm with 20-step Euler sampling and PickScore 27 as the objective, this process takes roughly hours on NVNIDA H100 GPUs
However, this cost is amortized over repeated use. Once set of heads is selected for given model and configuration, it can reused across different prompts and latent seeds-unlike test-time
scaling approaches 38 30) that require per-sample optimization. In the previous section, we further
13

---

Releiiee frequency
8.0%
Relellve Frequency
6.0%
6.0%
4.0%
4.0%
2.0%
2.0%
0.0%
0.0%
11 12 20
Layer Index
Layer
(4) Layer distribution of heads selected to improve general quality
(b) Layer distributiio of heads selected to improve style-orienteed quality
Toum::
teyed
Dount
fourd
14.0%
14.0%
5%
10
mm
10
12.9)
6.6
85
6.
6.0m
5.0
4.h
28%
35%
3.0
3.%
0.0%
Layer Bobe
Lager
Layer Indoo
Layer Vniles
Layer KdR
(c) Layer distribution of l'bcads selected for each round, to improve style-oriented quality
Figure 22: Layer distribution of heads selected by HeadHunter for different objectives. Effective heads are not confined to specific layers and the distribution is different with each objective, highlight- ing the limitations of layer-level guidance and the necessity of objective-specific head selection. (a) In general image quality setting, selections concentrate in early-mid layers (2.9% in layers 2 16). (b) In style-oriented quality setting, selections spread broadly, with 24.8% layers 2 (c) Round-wise distributions of (b) begin with an early-layer bias similar to (a), but gradually flatten over rounds, mirroring (b). This trend aligns with qualitative observations. Round primarily improves general quality, whereas stylization becomes more prominent in subsequent rounds.
demonstrate that head sets found using a small subset of prompts generalize well broader and more diverse prompt distributions. At inference time, attention perturbation guidance adds negligible overhead, as simply modifies fixed subset of attention maps without altering the model architecture or requiring additional forward passes.
14

---

No gailance
No gallance
Seyee
Style
panlpp obly
Siyle
Styde
(n) Unconditional generation results
(b) Some heads express style only using style prompts.
Figure 23: Investigation whether style enhancement occurs without style prompt when applying HeadHunter (style-oriented quality setting). Q indicates prompt and seed pairs. (a) In some cases, applying HeadHunter in extent. Style 1: "art nouveau style, elegant, decorative, curvilinear forms, nature-inspired, ornate, detailed".. Style 2: "line drawing, professional sleek modern minimalist graphic vector graphics", Style 3: "cubist abstraction, fragmented planes, bold geometric angles", Style 4: "cinematic lighting, dramatic tone". (b) In the other cases, style enhancement depends on the presence style prompts. For these heads, the stylization effect weakens without explicit style descriptions, highlighting the importance of including style prompts during the head selection process. Style prompt "psychedelic style, vibrant colors, swirling patterns, abstract forms, surreal, trippy, colorful reflective voids, perceptual disorientation, sinuous forms, color psychology"
unconditional setting amplifies the corresponding style to some
15

---

Baseline
r=2
r=
sunlit, warm glow, golden hour
Baseline
r=
r=3
r= r=4
spring bloom, fresh vibrant colors
Figure 24: Qualitative results of HcadHunter for style-oriented image quality improvement on SD3.
16

---

Baseline
r=
r=
r=3
r= r=4
cinematic lighting, dramatic tone
Baseline
r=2 r=1
r=3
r=4
r=
psychedelic style, vibrant colors, swirling patterns, abstract forms, surreal, trippy, colorful
Figure 25: Qualitative results of HeadHunter for style-oriented image quality improvement on SD3.
17

---

Baseline
r=2
r=3
r=
line art drawing, professional sleek
modern minimalist graphic vector graphics
Baseline
r=2
r=3
flat papercut style, silhouette, clean cuts, paper, sharp edges, minimalist, color block
Figure 26: Qualitative results of HeadHunter for style-oriented image quality improvement SD3.
18

---

Baseline
r=1
r=2
r=3
r=5 r=4
acid dreamscape, radiant color waves, warped geometry, cosmic hallucination effect
Baseline
r= r=
r=3
r=
cubist artwork, geometric shapes abstract innovative revolutionary
Figure 27: Qualitative results of HeadHunter for style-oriented image quality improvement I-Vx.
19

---

Baseline
r=1
r=2
r
r=
cyberpunk, neon lights, futuristic glow
Baseline
r=2
r=3
flat vector design, clean geometry, negative space, minimalist
Figure 28: Qualitative results of HcadHunter for style-oriented image quality improvement
I--v.
20

---

Baseline line
r=
r=2
r=3
r=5 r=4
spring bloom, fresh vibrant colors
Baseline line
r=1
r=2
r=3
r=s r=4
stained glass style, colorful glass fragments, lead came details, backlit, translucent, mosaic
Figure 29: Qualitative results of HeadHunter for style-oriented image quality improvement on FLUX.I-Dev.
21

---

E Additional results and analysis of head-level perturbation guidance
In this section, we provide an extended analysis of head-level perturbation guidance with a focus on interpretable visual concepts. Specifically, Sec. E.1 presents more individual head-level guidance results Stable Diffusion 3 and FLUX. 1-Dev. In Sec. E.2 we showcase examples of sead-level guid- ance that exhibit clearly interpretable semantic effects. Finally, Sec. explorees the compositional behavior of head-level guidance and how combining them yields richer visual outcomes.
22

---

E.1 Additional results of individual head-level guidance
Figure 30: Individual head-level guidance results on SD3, Each cell corresponds to the output guided by perturbing a single head. Some heads induce notable effects, such as changes in lighting. structure, or color.
23

---

Figure 31: Individual head-level guidance results on FLUX.1-Dew. Each cell corresponds to the output guided by perturbing a single head. Some heads induce notable effects, such as changes in lighting, structure, color.
24

---

E.2
Additional results on interpretable head-level guidance
Lo H2
LO
Peach pink
Red
L13
LO H4
Blurred brown
Sunlight
L1 H5
L14 H1
Linen canvas texture
Blazing colors
L14 H7
H14
High resolation, fine detailed
3D rendered style subject
L13 Hi2
H21
Simplifted detail & background
Cinematic style
L14 Hi.7
L15 H20
Rainbow neon filter
Vertical grain
L15 H3
L16H3
Over-contrast
Horizontal banding
L15 H23
L3 H19
Horizontal grain
Scan-line
L18 H18 H15
L18 H18
CGI render, high saturation
Vignetting, Digital noise
L10 L10 H12
L17 H22
Natural light, shallow focus
Amorphous art, flat shading
Figure 32: Interpretable heads in SD3. Some heads exhibit consistent and clear visual effects when used for guidance. For instance, heads that add glow, alter geometry, introduce specific lighting styles. These intcrpretable heads support the view that attention heads encode specific semantic concepts.
25

---

L34 HO
HB
Smooth bokeh, detailed
Natural light photorealistic
L43 H117
L51 H23
Glossy, plastic texture
Vintage film style
L35 H16
138
Durk cinematic style
Golden hour
L36 H2
L51
Moody, low-key lighting
RGB-split
LS4 H23
LS2 H22
Autumnal cinematic style
Warm red palette
152 H12
L52 H6
Cool
Green filter
Figure 33: Interpretable heads in FLUX.I-Dev. Similar to SD3, FLUX also contains heads with distinctive effects, suggesting the generality of head-level interpretability across architectures.
26

---

E.3
Additional results on the combinational effect of head-level guidance
{LO H3, L16 H3)
Red Murizuntal bunding
Horizontal bunding
{LO H4,
Blurred brown
L13 H18)
Blured brown Sunlight
Suotigght
(L14 H7,
High resolation fine detailed
L14 H17}
High resolation, fine detailed Rainbow neon filter
Rainbow neow filee
{LO H3, H18)
Red
Rel Bluzing cofors
Blazing colors
Figure 34: Compositional effects of head-level guidance SD3. Combining heads leads to enriched outputs by blending their individual effects. This shows that attention heads can be composed to control more complex or stylized generations.
F Head-level perturbation guidance with different perturbation methods
In the main paper, we show the head-level analysis mostly on identity-matrix matrix replacement perturbation. In Fig. 37 We show the results with another perturbation and the analysis. Note that one can freely choose any other perturbation methods, and we show some cases as examples.
27

---

(L51 H23,
Vintage film style
L52 H22)
Vintage film style Warm red palette
Warm red palette
HO,
Smooth bokeh. detailed
H6}
Smooth bokeh, detailed + Cool tone
Cool tone
{L38 H18,
Golden hour
LS2 H12)
Golden hour
Green filter
Green filter
(L38 H18,
Golden hour
L43 H17)
Golden hour Glossy, plastlc texture
Glossy, plastic texture
Figure 35:
Compositional effects of head-level guidance in FLUX.1-Dev. Similar head-level guidance effects can be composed in FLUX.I-Des to amplify or adjust stylistic elements.
28

---

(L1 H2,
Dork
L1 H10))
Darker
Dark
H15, H19)
White
More white
White
(u) Individual head perturbation
(b) Multiple heads perturbation
Figure 36: Concept amplification by combining stylistic heads. Composing "dark heads" makes the generation darker, while "white heads" make it brighter. This demonstrates fine-grained control over stylistic dimensions via head-level selection.
L12 H6
ETIL
Soft shudow
High
L13 H7
LI5 H7
Stollight
Shallow focus, Photorealism
L19 H13
L18 H16
PHS Scan-line
Background simplified
L17 H17
L17
High saturation, Digital illustration
Horizontal grain
L10 H19
L19 H17
Photorealism
Linen carvas texture
(a) Unifomm Guidance
(b) Smoothed Energy Guidance
Figure 37: Head-Level perturbation guidance with different perturbations. We show interpretable heads with different perturbation methods.
29

---

G
Experimental details
Unless otherwise specified, all images are generated using Stable Diffusion 3-Medium with 20 Euler sampling steps. The default guidance scale is set to w 5.0.
Fig. Motivating examples for head-level guidance. Perturbations are applied individually to heads from layers 1, 3, and 14. Prompts are sampled from the MS-COCO validation set.
Fig. Generated images from head- and layer-level perturbation guidance. In (a), heads with PickScore above fixed threshold are selected (indicated by red boxes). Prompts include "a medieval castle in the forest" and "a futuristic cityscape".
Fig. Effect of head-level guidance on concept amplification. Each head is perturbed inde- pendently. Prompts used include: "smiling girl holding a cat, a flower garden", " "middle-aged woman with glasses reading a book in a café", "A macro shot of red apple". close up portrait of an elderly Japanese man, soft lighting, 8k", and "A German shepherd bounding through a pine forest". Guidance scale is set to w 5.0.
Fig. Results of HeadHunter for general image quality improvement. HeadHunter applied with R = and k 24. Prompts used include "smiling girl holding cat, in flower garden". "businesssan with sleek, glass-shiny black hair neatly parted, waiting for a morning train on a city platform', and "towering lighthouse casting rotating beam across storm-tossed waves at twilight', Metrics such as PickScore and AES are used for evaluation.
Linear interpolation between attention map A and identity matrix (SoftPAG), Images are generated with perturbation applied to layer Prompts are generated from ChatGPT.
Fig. 9
Fig. 14 Linear interpolation between attention map A and uniform matrix U (UG). Perturba- tion is applied to layer 9, Prompts are the same as those used in Fig.
H Experimental resources
All experiments, including testing perturbation methods, analyzing head-level guidance, running HeadHunter, and conducting ablation studies, are performed using a mix of 8 NVIDIA H100 GPUs, 6 NVIDIA RTX 3090 GPUs, and 2 NVIDIA A6000 GPUs.
I Limitations & Broader Impacts
While our method is effective, there remains room for improving its efficiency. Running HeadHunter iteratively can computationally intensive, especially for large-scale models with many heads. That said, the selected heads tend to generalize well across prompts and latents, making reuse practical. Exploring faster head and parameter selection strategies offers an exciting direction for future work.
Our work improves quality and stylistic control in diffusion models through guidance. This can benefit creative applications such digital art, design. However, enhanced quality may also increase the risk of misuse, including the creation of deceptive or harmful content like deepfakes. While our work does not involve identity synthesis model release, we acknowledge this risk and recommend responsible deployment practices such as usage restrictions and content disclosure.
30

---

Table 3: Prompt and seed list used for general quality improvement via HeadHunter.
Prompt
Seed
close up of an old Japanese man, soft lighting, 8k
0
A black man with long braids, wearing white robes and heavy metal jeweiry, pointing at the
0
viewer against dark background the style of cinematic photography for fashion shoot.
A photo of Asian girl with her hand her nose, with red nail polish and bandaid, with black hair, close up portrait, in the style of Rinko Kawauchi.
0
Japanese woman with short black hair, bangs hairstyle, hand covering eyc, ring rings on, cool pose, outdoor sunlight background, natural lighting, portrait photography, in the style of Leica Q2 camera
A group of models of diverse ethnicities wearing various Nike sneakers, all dressed in white 0 and colorful with patterns and designs inspired by the pop art movement, posed together for an editorial photoshoot against backdrop featuring abstract collage-like elements in shades of red and blue. The scene was vibrant and dynamic, capturing their energetic expressions and stylish outfits in the style of the pop movement.
photo of black special forces soldier doing an olympic ski jump, holding two katanas in his hands, white background, 35mm film stills the style of Cai Guo-Qiang and James
A tall German man sitting in bench, reading book, medium shot
0
A breakdancer doing a backflip
0
A photograph of an old oak tree in the middle, surrounded by dense woodland foliage, with sunlight filtering through the leaves and casting dappled light on the ground. The image was shot using a Hasselblad camera, Kodak Gold 200 film, and Portra 800 film stock. a close-up of woman's s hand with delicate fingers, soft lighting, photorealistic a photo of beautiful girl
0
0-9
Table 4: Content prompts and corresponding seeds used for style-oriented quality improvement via Header
Prompt
Seed
portrait of an elderly man with white hair, wearing a wool coat, looking into the distance
0
young woman in red dress, standing in the wind, eyes closed
teenage boy with messy hair, wearing headphones, sitting on rooftop
2
smiling girl holding cat, in flower garden a man leaning against a brick wall, hands pockets, calmly observing the street
3
4
31


Reviews:
[{"review_id": "I2TiPfqLiN", "reviewer": "Reviewer_X5tq", "summary": "This paper introduced a unified framework for self-attention score guidance. Basically ablating certain attention head (interpolating between original and identity or flat map) and making them negative control, one can do classifier free guidance with itself and achieve better quality or style. The authors systematically studied the effect of perturbing different attention heads and combination of them. Finally they developed a pipeline to systematically search for heads combinations to optimize certain objective like style.", "questions": "- What is the complexity of the head hunter method? It seems need to run through all the layer x head and all the prompt once per iteration, that is hundreds of heads and runs. So in a sense it’s very expensive to run this head hunter once. Then the question becomes how useful and generalizable it is? Can the authors show that they can run it once for some prompts and then generalize to many more new prompts / context?\n- As follow up, If so, isn’t the better thing to do is to finetune the model or perform some PEFT specific to those heads? How does the HeadHunter compare to just finetuning?\n- The paper introduced the additional $u$ parameter, but could it be equivalent to tune the cfg guidance scale parameter $w$ for previous methods? \nThis is esp. concerning for the Figure 6 head vs layer comparison, it seems likely that the layer result looks bad because the guidance is too strong….and it’s intuitive that if all attn head in a layer was perturbed, then the diff between two runs will be greater, so the guidance strength should be tune down for it… The authors also noted themselves, at L216, the more head is perturbed, the more likely it is to oversaturate. \n Is there an experiment showing tuning the $u$ parameter is more than tuning guidance scale?\n- In section 3, you introduced an additional interpolation parameter $u$, on the bright side, we have another knob to tune; on the dark side, we have yet another hyperparameter to tune. Is there any optimal choice of this value that could be transferred to new experiments? or does it have high variability from head to head and need to tune separately? Do you need to tune u for each head in HeadHunter?\n- (Minor) L131 Why 2D Gaussian blur cannot be applied to modern MMDiT architecture?", "limitations": "Yes in appendix H.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strength\n\n- The method is simple to implement and quite principled. The introduction of this method as a conceptual unification of previous guiding method (PAG, SEG) is nice.\n- The systematic study and description of perturbation effect of different head and combination of them is quite interesting (Fig.4,5). It’s thought-provoking to the field for interpretability study in diffusion model.\n- Localizing perturbation to head is interesting and illuminating, from Fig6 seems like it’s a strong and targeted effect.\n\n### Weakness\n- Lack of theoretical framework and further understanding of the observed effect. For example, authors could discuss\n    - it’s a bit unclear if the effect of perturbing a head is equivalent to perturb a layer with less strength $w$. (see questions)\n    - Or perturbing / ablating the attention map in the negative run is equivalent to enhance the map in the positive run.\n- The proposed HeadHunter iterative method is a great proof of concept but it seems a bit *ad hoc,* and expensive… not sure if it’s actually practical. (see questions)\n- Minor: The main text may benefit from a slightly more detailed description of the HeadHunter method, currently all the relevant details are buried in appendix D.", "was_revised": true, "final_justification": "The authors have addressed all our concerns raised in our review! We decided to raise the score to 4. \nMain reason for increasing the score is the explanation and new experiments for 1) method efficiency, 2) the generalizability of the method and 3) comparing it with PEFT makes the method more impactful! \nThese evidence convince me that the proposed method has some potential for practical use.  \n\nThe new results of tuning the cfg scale for layer vs single head is very interesting. I feel the inital concern (*equivalence between disrupting one head and disrupting a layer with smaller cfg*) is sometimes valid, but definitely not for every layer. The authors are very transparent about it and we appreciate their honesty", "rebuttal": "We are grateful to the reviewer for their time and insightful feedback, which helps us a lot to improve our work. We address the questions:\n\n> **[Q1]** What is the complexity of the head hunter method?\n> \n\n**[A1]** Thank you for the question. As noted in Appendix D.3.2, we consider following parameters to compute NFE :\n\n- $T$: Total search iteration\n- $M$: Number of prompt-seed pairs\n- $N$: Number of total attention heads in model\n- $S$: Number of denoising steps\n\nIn total, our framework requires $T \\cdot M \\cdot N \\cdot (2S+1)$, considering two forward pass for guidance and one NFE for verifier.\n\nIn our general quality improvement setting, we use $M = 20$, $T = 1$, and in the style-oriented setting, $M = 5$, $T = 5$. Both use $N = 24 \\times 24$ heads and $S = 20$ steps. On an 8×H100 GPU node, each iteration takes roughly 36 minutes.\n\nImportantly, this is a one-time cost, which can be amortized over many prompts thanks to the strong *generalizability* of the selected heads. We elaborate on this in [A2].\n\n> **[W2 Q1]** How useful and generalizable it is? Can the authors show that they can run it once for some prompts and then generalize to many more new prompts / context?\n> \n\n**[A2]** In Fig. 7(b), we show that heads selected for improving general image quality using only 20 prompts also enhance the quality of samples from **unseen prompts** in the MS COCO validation set.\n\nFor style-oriented improvement, we demonstrate in Appendix D.3.2 (*“Generalization across content prompts”*) that head sets selected from just 5 content prompts can improve style fidelity for 50 **novel prompts** generated by GPT-4o. Fig. 26 and Tab. 2 report both qualitative and quantitative results, confirming HeadHunter's strong generalization to new prompts.\n\n> **[Q2]** Isn’t the better thing to do is to finetune the model or perform some PEFT specific to those heads? How does the HeadHunter compare to just finetuning?\n> \n\n**[A3]** Thank you for the insightful question. To investigate this, we ran preliminary experiments fine-tuning LoRA modules on the interpretable heads identified in Fig. 5 and Fig. 22 (e.g., “darkness,” “shadow,” “blue,” “sunlit”). For each head, we generated samples using perturbation guidance with related prompts and then trained LoRA on the query/key/value projections of that head both with and without out projection. We compared the results with those from LoRA fine-tuning and full attention layers.\n\nInterestingly, we found that fine-tuning on the retrieved heads does not consistently lead to improved style alignment. In fact, training on HeadHunter-retrieved heads resulted in *weaker* alignment to the intended style than training on full layers (see Fig. 8). We hypothesize that this is because the retrieved heads already encode the desired style, so further tuning provides little benefit. In contrast, unrelated heads have more room to adapt, leading to stronger optimization. \n\n> **[W2-2]** Perturbing / ablating the attention map in the negative run is equivalent to enhance the map in the positive run.\n> \n\n**[A4]** Thank you for the comment. As noted in autoguidance [1], \"*it is easy to get a worse version of the model, but much harder to make it better*.\" and principled way to enhance the map in positive run does not exist as far as we know. \n\nHowever, we tried to explore an alternative perspective to enhance the effects of attention perturbation on positive run. One possible way could be the method of Prompt-to-Prompt [2], where the attention maps are amplified by multiplying them with a scalar greater than one. We conducted such experiments using interpretable heads identified in Fig. 5 and Fig. 22 (e.g., the “darkness”, “shadow”, “blue”, and “sunlit” heads) both with and without prompts containing the relevant concepts.\n\nOur results showed that boosting the corresponding heads can emphasize their associated concepts to some extent. For example, enhancing the “darkness” head produced thicker and more pronounced shadows; the “sunlit” head added yellow highlights near human contours and the “blue” head introduced blue elements in headphones or backgrounds. However, we also observed that this often came at the cost of **greatly reduced image quality with notable artifacts.**\n\nThese findings suggest that while concept-specific heads can indeed be identified and amplified, directly enhancing attention maps can be hard to achieve and unstable. In this context, using negative guidance offers a more practical alternative for emphasizing desired styles or attributes.\n\n> **[W1]** it’s a bit unclear if the effect of perturbing a head is equivalent to perturb a layer with less strength $w$. (see questions) \n**[Q3]** Could it be equivalent to tune the cfg guidance scale parameter $w$ for previous methods? … Is there an experiment showing tuning the parameter is more than tuning guidance scale?\n> \n\n**[A5]** Thank you for raising this important point. We would like to clarify that perturbing a single head is not equivalent to perturbing an entire layer with a lower guidance scale $w$.  As a layer is equivalent to the multiple heads (24 heads in SD3), let's think about the single-head vs two-heads case. In Fig. 5, we showed that applying head-level guidance for each head results in a distinct visual attribute, and the case of two heads results in the composition of the concept implied by each head. If the reviewer’s assumption is true (layer perturbation with a lower guidance scale is equivalent to fewer head perturbations), the effect of reducing the strength across multiple heads should be the same as one of the concepts in each head. To check this, we conducted several experiments using the attention heads in Fig. 5(b).  Applying head-level guidance with  “blue” and “dark” heads with lower guidance scale did not correspond to either the “blue” head perturbation alone or “dark” head perturbation alone. But it exhibited the diluted “dark blue” concept and the results were similar for the case of other head combinations.\n\nAlso due to the difference of head number, we agree that the guidance strength of Fig. 6 (b) may have been too strong, leading to unfair comparison. As suggested, we re-evaluated with lower guidance scales and reported the quantitative comparison using PickScore below. (Due to the rebuttal constraint, we only provide quantitative results, will include sample images in the revised manuscript.) \n\nFor L3 and L8, your comment was correct. We confirmed that reducing guidance at the layer level yielded performance comparable to head-level guidance. As these are not suitable examples, we will revise the manuscript accordingly, and we are grateful for your valuable feedback in improving the paper.\n\nHowever, we would like to respectfully draw the reviewer’s attention to L12 and L13. Despite using a very small guidance scale, L12 still fails to surpass the performance of head-level guidance. L13 even consistently underperformed compared to the baseline (unguided) across all guidance scales. These results still make our analysis valid that even for underperforming layers, selecting the right heads can yield strong results.\n\n**PickScore comparison**\n\n| Layer / $w$ | Baseline (Unguided) | 1 | 2 | 3 | 4 | 5 | Ours (Head-level Guidance, w=5.0) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| 3 | 21.08 | 22.07 | **21.87** | 21.43 | 17.67 | 18.36 | 21.37 |\n| 8 | 21.08 | 22.31 | **22.43** | 20.94 | 20.36 | 21.41 | 22.23 |\n| 12 | 21.08 | 21.77 | 21.51 | 20.7 | 20.44 | 17.81 | **22.52** |\n| 13 | 21.08 | 19.62 | 19.12 | 17.39 | 16.55 | 16.75 | **22.47** |\n\n> **[Q4]** Choice of hyperparameter $u$\n> \n\n**[A6]** Thank you for the great question. We found that using a fixed guidance scale of $w=3.0$ and interpolation parameter $u=1.0$ works well across different style prompts in the style-oriented quality improvement setting. This fixed configuration was used throughout our main experiments (Fig. 8 and Fig. 9) across 23 style prompts, and also in the generalization test (Fig. 26, Tab. 2), demonstrating strong qualitative and quantitative performance.\n\nAs the reviewer suggested, it is indeed possible to tune $u$ individually for each head to potentially improve quality further. However, we found that applying the same $u$ to all selected heads yields sufficiently strong results in practice. In our HeadHunter search phase, we set $u=1.0$ to maximize the influence of each head in expressing a specific style. For inference-time usage, users may optionally reduce $u$ if the perturbation appears too strong.\n\nImportantly, our method is not restricted to a specific value or search strategy for $u$. The choice of $u=1.0$ was based on empirical effectiveness and simplicity, but HeadHunter remains compatible with alternative tuning schemes if desired.\n\n> **[Q5]** (Minor) L131 Why 2D Gaussian blur cannot be applied to modern MMDiT architecture?\n> \n\n**[A7]** Thank you for the opportunity to clarify. In MMDiT architectures, attention maps span both 2D image tokens and 1D text tokens. Therefore, the original paper’s 2D Gaussian blur to the full attention map is not feasible. While we experimented with applying the blur only to the image attention submatrix, it led to degraded performance and unintended artifacts. Moreover, such partial perturbations do not generalize well to multi-modal models (e.g., video), where attention spans additional dimensions like time, and would require heuristic designs that we believe are not scalable.\n\n> **[W3]** Minor: The main text may benefit from a slightly more detailed description of the HeadHunter method, currently all the relevant details are buried in appendix D.\n> \n\n**[A8]** Thank you for suggesting this. We will add a more detailed description and the algorithm of the HeadHunter method so that readers can grasp the idea more clearly in our manuscript. \n\n---\n\n[1] Guiding a diffusion model with a bad version of itself, NeurIPS 2024\n\n[2] Prompt-to-Prompt Image Editing with Cross Attention, arxiv 2022"}, {"review_id": "YC7wiJfzSC", "reviewer": "Reviewer_ZmWS", "summary": "Based on current guidance methods for diffusion models, the authors propose a unified attention perturbation guidance model, which interpolates between the original attention map and a perturbed attention map. Specifically, the authors interpret the attention maps as a set of probability distributions, where the uniform distribution represents the scenario with the highest entropy, and the identity matrix can be interpreted as the lowest entropy case. Interpolating between the extremes is investigated experimentally in the first half of the paper, and it is concluded that there exists a certain trade-off. In the second half of the paper, the authors investigate the impact of different layers and attention heads on the guidance methods and propose an iterative framework, called Headhunter, for identifying attention heads that are interpretable and composable.", "questions": "- How does perturbation guidance perform compared to regular classifier-free guidance or auto-guidance? \n- How should perturbation guidance be interpreted? How does it affect the estimated probability density path?\n- Are the effects observed for each attention head only valid for attention perturbation, or could it also be used independently? If so, how does it perform compared to the proposed method?", "limitations": "Yes (in the Supplementary Material)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Interesting results and thorough investigation\nThe paper goes into great detail investigating the effects of attention-based perturbation guidance on a layer and attention-head level. For the generalized framework of attention perturbation guidance, the authors carefully explore the variety of hyperparameters such as the interpolation coefficient, the guidance strength or the target layers or attention heads. It is interesting to see that selectively guiding certain attention heads impacts specific properties and can be used for controllability purposes.\n\n- Automated framework appears promising\nThe proposed HeadHunter framework allows for a principled and automated approach to find relevant attention heads that improve the generation quality of the image under a predetermined objective function. By composing multiple of these heads, the overall performance can be improved over the unconditional case.\n\n\n### Weaknesses\n- Probability interpretation is mathematically questionable/not well-motivated\nThe interpretation of the attention maps as a set of probability distributions is not well defined in the paper and might be confusing for some readers, as it might imply some stochastic modelling of the attention values. Interpreting the values of the attention map as a set of probability distributions appears to only matter for the argument of minimal/maximal entropy scenarios, which itself does not provide much additional insight into the method. For the minimal entropy scenario, there exists more than just the identity matrix option, which is referred to as 'max guidance' in the Supplementary Material. Technically, the interpolation could be performed on the entire probability simplex, which offers a large number of options.\n\n- Unclear Figures and parameter settings\nThroughout the paper, it remains unclear what value is used for the interpolation parameter u or the guidance scale w in the figures for a lot of scenarios. This makes it hard to understand the exact effects of each parameter and the generalizability of the method.\n\n- Perturbation mechanism remains unclear, mostly empirical observations\nWhile the paper provides a wide variety of different experimental results, it lacks an answer to the deeper mechanisms of the perturbation mechanism. For classifier-free guidance, the intuition behind weighing the conditional signal more strongly stems from sharpening the estimated conditional distribution, but in the case of perturbation guidance, no intuitive interpretation is given.", "was_revised": true, "final_justification": "After the discussion with the authors, most of my questions have been answered, and concerns were addressed. While the findings presented in the paper are interesting and noteworthy, the unclear description and settings of the method hinder a deeper understanding to the point where reproducibility becomes a concern. As a result, I decided to keep my borderline accecptance rating.", "rebuttal": "We sincerely appreciate your careful evaluation of our manuscript, which helps us a lot to improve our work. We address your questions as follows.\n\n> **[W1]** Mathematical soundness/motivation for interpreting attention maps as a set of probability distributions\n> \n\n**[A1]** Thank you for the insightful comments. The original attention mechanism paper [1] interprets attention weights as probabilities of alignment between target and source tokens, and subsequent works have adopted a similar probabilistic interpretation. We extend this view to attention maps to unify existing guidance methods and to enable a natural formulation of interpolation between distributions.\n\nAlthough interpolating two distributions is intuitive and natural action in the probability distribution space, this idea has been largely overlooked in prior work, and the introduction of the probability-space perspective *allowed us to introduce **interpolation*** between the original and perturbed attention maps.\n\nWhile the probabilistic interpretation supports a unified view of existing guidance methods and motivates the introduction of interpolation, we acknowledge that entropy-based explanations may not provide substantial additional insight and could cause confusion. Therefore, we will minimize the discussion of probabilistic, especially entropy-based views in the revised manuscript, and instead place greater emphasis on interpolation, which plays a key role in controlling perturbation strength.\n\n> **[W1]** Interpolation on other paths.\n\n**[A2]** As the reviewer pointed out, alternative interpolation paths in the probability space, such as Fisher-Rao interpolation, can lead to slightly better image quality. We present the results in the table below:\n\n| *Interpolation Method /* $u$ | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| Linear | 20.65 | 22.15 | 22.32 | 22.32 | 22.21 | 21.90 |\n| Fisher-Rao | 20.65 | 21.80 | 22.34 | 22.30 | 22.03 | 21.90 |\n\nWe can see that Fisher-Rao interpolation provides slightly higher scores at certain $u$ values than linear interpolation. However, they also complicate the implementation and may introduce numerical instability. In addition, we initially attempted to implement optimal transport (OT) geodesic interpolation, but found it difficult to implement efficiently.  Given its simplicity, stability, and competitive performance, we adopt linear interpolation in our main experiments. Nonetheless, users may still consider using more complex paths if slightly better quality is desired. We will include these results and the discussion in the revised manuscript.\n\n> **[Q1]** How does perturbation guidance perform compared to regular classifier-free guidance or auto-guidance?\n\n**[A3]** We kindly refer the reviewer to [A4] for a conceptual comparison between perturbation guidance, CFG, and Autoguidance. In this response, we interpret the question as a request for a performance comparison.\n\nOur goal is to improve attention perturbation guidance through fine-grained control and principled objective-driven head selection, rather than to outperform CFG or autoguidance. Nonetheless, for reference, we conducted experiments on 1K prompts from the MS-COCO validation set and report the results below. For perturbation guidance, we use the top-k heads as described in Sec. 4.2.1.\n\n|  | FID | IS | Precision | Recall | Density | Coverage | PickScore | ImageReward |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Baseline | 64.08 | 18.61 | 0.44 | 0.56 | 0.53 | 0.04 | 20.61 | -0.31 |\n| CFG 4.0 | 54.84 | 25.45 | 0.71 | 0.48 | 1.23 | 0.08 | 22.38 | 0.91 |\n| Autoguidance ($w$=4.0, $D_1$: dropout with p=0.001) | 68.49 | 18.94 | 0.43 | 0.40 | 0.42 | 0.03 | 20.86 | -0.09 |\n| Ours (top-k=21, $w$=4.0, $u$=0.5) | 56.85 | 24.18 | 0.65 | 0.44 | 1.02 | 0.06 | 22.32 | 0.73 |\n| Layer 9 ($w$=4.0, $u$=0.8) | 59.23 | 24.32 | 0.63 | 0.43 | 0.97 | 0.06 | 22.28 | 0.71 |\n\nWe note that prior perturbation guidance methods (PAG and SEG) were also not designed to outperform CFG. In general, perturbation guidance offers unique advantages, including compatibility with unconditional generation and better preservation of sample diversity. Moreover, it can be combined with CFG to leverage the strengths of both approaches. We demonstrate this in Tab. 3 and Fig. 27 in the Appendix.\n\nIn this work, we focused on improving perturbation guidance by introducing head-level control. As shown in the last two rows of the table, our best-performing head-level configuration outperforms the best-performing layer-level setting (which corresponds to PAG with principled layer search) across nearly all metrics. This suggests that head-level perturbation offers more effective and fine-grained control compared to layer-level approaches.  The same trend holds for style-oriented quality improvement, as shown in the table provided in response [A1] to reviewer xdij.\n\n> **[W3]** Deeper mechanisms of the perturbation mechanism. Intuitive interpretation like CFG.\n**[Q2]** How should perturbation guidance be interpreted?\n\n**[A4]** Thank you for raising this important question. We adopt the interpretation proposed in autoguidance [2] to help explain how perturbation guidance works. [2] argues that weakening the model results in a smoother sample distribution that includes outlier or erroneous regions. By guiding away from the weaker model, Autoguidance avoids these undesirable regions and helps correct the model’s errors.\n\nPerturbation guidance operates in a similar manner. By perturbing the original model’s attention distribution, we create a degraded variant that captures the erroneous regions. The difference between the original and perturbed outputs provides a meaningful guiding signal that can steer the sampling process away from those regions.\n\nIn this framework:\n\n- The **perturbation method** defines *how* the sample distribution is perturbed (i.e., which regions of the distribution are smoothed) and thus determines the **direction** of the guidance signal.\n- The **perturbation strength** defines *how much* the distribution is perturbed (i.e., the degree of difference between the original and perturbed scores) and also affects the **direction**.\n- The **guidance scale** $w$, in contrast, controls *how far* the sample moves in that direction and affects only the **magnitude** of the signal, not its direction.\n\nTherefore, the perturbation method/strength plays a crucial role in shifting the sampling distribution to cover outliers/erroneous samples while avoiding overly-smoothed distribution, which would lead to ineffective guidance. This intuition highlights the importance of selecting an appropriate perturbation strength, as also observed in Autoguidance, where slightly degraded models (e.g., S vs. XS) yield the best performance.\n\nSimilarly, in our setting, the perturbed attention heads determine the shape of the smoothed distribution by selecting which visual attributes to perturb, the direction of perturbation, and the interpolation parameter $u$ controls its strength. Together, they determine the direction of the guidance signal. This is fundamentally different from tuning the guidance scale $w$, which only adjusts the magnitude. This insight highlights the importance of controlling $u$.\n\nIndeed, we conducted experiments where we reduced the guidance scale. The best results were consistently achieved by tuning $u$, as shown in the table below:\n\n**PickScore comparison**\n\n| $w$ / $u$ | 0.25 | 0.5 | 0.75 | 1.0 |\n| --- | --- | --- | --- | --- |\n| 2.0 | 22.07 | 22.28 | 22.19 | 21.92 |\n| 4.0 | **22.31** | 22.29 | 22.08 | 21.66 |\n| 6.0 | 22.29 | 22.12 | 21.78 | 21.27 |\n\n> **[Q3]** Are the effects observed for each attention head used independently? If so, how does it perform compared to the proposed method?\n\n**[A5]** Thank you for the thoughtful question. We refer the reviewer to response [A3] to Reviewer X5tq for our preliminary exploration of parameter-efficient fine-tuning (PEFT) applied to specific heads retrieved by HeadHunter. We observed that fine-tuning individual heads can lead to different visual outcomes, suggesting that attention heads may encode distinct visual semantics.\n\nHowever, we did not observe a clear correlation between the heads selected by HeadHunter and improvements in PEFT efficiency or style fidelity. Our hypothesis is that heads already specialized for certain styles may not benefit from further training, whereas other heads that are less aligned with the target style may still have capacity to learn, resulting in greater improvement when trained.\n\nAdditionally, in this response, we explore an alternative approach where we directly scale the attention maps of selected heads using a fixed multiplier, inspired by Prompt-to-Prompt [3]. We apply this to interpretable heads we found (e.g., “sunlit,” “shadow,” “blue”). Interestingly, this method does enhance certain visual concepts consistent with those observed under perturbation guidance, for instance, increasing shadow intensity or emphasizing specific colors.\n\nHowever, we also found that this approach frequently introduces noticeable artifacts and tends to degrade overall image quality, making it less practical in real applications.\n\nIn general, it is considerably easier to degrade model performance than to improve it. This aligns with findings from AutoGuidance [2] and supports our claim that HeadHunter’s negative guidance strategy provides a more stable and effective direction for controllable generation.\n\n> **[W2]** Clarification on parameter choice.\n\n**[A6]** We apologize for the confusion. In Sec. 3 and Sec. 4, we used $w = 5.0$ and $u = 1.0$. For HeadHunter, we used $w = 3.0$ and $u = 1.0$, except in Fig. 7(b), where $u = 0.25$. This follows standard practice in diffusion literature [4–7], which uses different settings for quantitative and qualitative comparisons. We will clearly state all parameter values in the manuscript.\n\n---"}, {"review_id": "iDW0y5TzwU", "reviewer": "Reviewer_ZVaq", "summary": "This paper propose a method for implementing finer-grained perturbations on modern large-scale text-to-image diffusion and flow models. Existing methods typically perturb at the layer level, this paper targets the perturbation precisely to individual attention heads (HeadHunter). The experiments show that precise control of specific visual attributes (e.g., concept, style, image quality) in the generated image can be achieved by perturbing specific attention heads. This approach overcomes the side effects such as over-smoothing or over-saturation that may result from existing perturbations, and to provide a plug-and-play solution that enhances existing inference pipelines without requiring retraining.", "questions": "1. This paper proposes to view attention weights as probability distributions and to perform interpolation, which indeed offers a more unified and intuitive perspective for understanding PAG and SEG. However, this \"unification\" primarily refers to linear interpolation. The paper mentions the existence of more complex paths between probability distributions (e.g., Fisher-Rao or optimal transport geodesics) , whether these more complex geometric structures could lead to superior perturbation strategies? Or why simple linear interpolation proves to be highly effective in mathematical principles?\n2. The basis for the selection of the parameters “number of heads selected per round n” and “maximum number of iterations T” is not specified. How do these parameters affect search efficiency and final results, and are there smarter search strategies (e.g., based on gradients or interpretability scores) that can replace or complement this trial-and-error generation-evaluation cycle?\n3. This paper notes that “adding more perturbed heads results in over perturbation”, which limits the number of heads that can be combined. HeadHunter, while finding good combinations, does not explicitly give a strategy for how to automatically determine the optimal number of combinations (i.e., when to stop adding heads), and still relies on observing performance curves. Does this rely too much on manual evaluation and weaken its generalization?\n4. In the appendix, each concept is associated with different heads of different layers (e.g. Dark and White), and in practice, when users want to optimize certain styles, they need to train to find the corresponding layers and heads first. However, users applications that require rapid iteration, this strategy of iteratively generating and evaluating each individual head can become computationally prohibitive. How about the computational efficiency and scalability?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. Experimental designs and analysis are good.\n2. The method can enhances existing inference pipelines without requiring retraining, it will be highly practical and scalable, and can be applied directly to existing pre-trained large diffusion models (e.g., Stable Diffusion 3 and FLUX).\n3. Head-level guidance would allow the user to perform highly precise and targeted manipulation of specific visual attributes of the resulting image.\n\nWeaknesses:\nThe claim made by the work provides a simple approach to finer-graining target style or concept without training and is, in general, innovative. There are sufficient visual results to support the claim. However, the explanations of why some strategies were chosen are relatively weak, with unclear parameter selection. For example, the interpolated parameter u provides intuitive control, but its theoretical optimality is not sufficiently analyzed. A “sweet spot” region is found experimentally, but this is more of an empirical discovery than based on theoretical derivation.\n\n(More specified in the “Question”).", "was_revised": true, "final_justification": "The proposed method is innovative, and the authors' rebuttal successfully justifies their parameter choices and the methodology's rationale. However, as noted previously, the paper's theoretical underpinnings remain weak. Additionally, it is debatable whether the slight performance improvement in certain scenarios justifies the increased complexity from the large number of heads. For these reasons, I will maintain my original score as a borderline accept.", "rebuttal": "We sincerely appreciate your careful evaluation of our manuscript, which helps us a lot to improve our work. We address your questions as follows.\n\n> **[W1]** The interpolated parameter u provides intuitive control, but its theoretical optimality is not sufficiently analyzed. A “sweet spot” region is found experimentally, but this is more of an empirical discovery than based on theoretical derivation.\n> \n\n**[A1]** We’d like to offer a theoretical explanation for why a “sweet spot” emerges, drawing on the framework introduced in autoguidance [1]. The autoguidance paper argues that weakening the model results in a smoother sample distribution that includes outlier or erroneous regions. By guiding away from the weaker model, Autoguidance avoids these undesirable regions and helps correct the model’s errors.\n\nPerturbation guidance operates in a similar manner. By perturbing the original model’s attention distribution, we create a degraded variant that captures the erroneous regions. The difference between the original and perturbed outputs provides a meaningful guiding signal that can steer the sampling process away from those regions.\n\nIn this framework:\n\n- The **perturbation method** defines *how* the sample distribution is perturbed (i.e., which regions of the distribution are smoothed) and thus determines the **direction** of the guidance signal.\n- The **perturbation strength** defines *how much* the distribution is perturbed (i.e., the degree of difference between the original and perturbed scores) and also affects the **direction**.\n- The **guidance scale** $w$, in contrast, controls *how far* the sample moves in that direction and affects only the **magnitude** of the signal, not its direction.\n\nTherefore, the perturbation method/strength plays a crucial role in shifting the sampling distribution to cover outliers/erroneous samples while avoiding overly-smoothed distribution, which would lead to ineffective guidance. This intuition highlights the importance of selecting an appropriate perturbation strength, as also observed in autoguidance, where slightly degraded models (e.g., S vs. XS) yield the best performance.\n\nIn our setting, the **selection of attention heads** defines the **perturbation method** (which visual attributes to perturb), and the **interpolation coefficient** $u$ controls the **strength** of the perturbation. Together, they determine the final guidance signal. This mechanism is fundamentally different from adjusting the **guidance scale** $w$, which only affects the signal's magnitude.  This insight highlights the importance of controlling $u$. As shown below, tuning $u$ consistently yields better results:\n\n**PickScore comparison**\n\n| $w$ / $u$ | 0.25 | 0.5 | 0.75 | 1.0 |\n| --- | --- | --- | --- | --- |\n| 2.0 | 22.07 | **22.28** | 22.19 | 21.92 |\n| 4.0 | **22.31** | 22.29 | 22.08 | 21.66 |\n| 6.0 | **22.29** | 22.12 | 21.78 | 21.27 |\n\nWhile we provide theoretical reasoning for the existence of such a sweet spot, we acknowledge that the precise optimal point is determined empirically. This is consistent with other methods: for example, AutoGuidance does not analytically derive an optimal weakening strength (e.g., via model size, EMA length, or training time), and the original CFG paper [2] offers a theoretical motivation for combining conditional and unconditional distributions, but identifies the optimal scale $w$ through empirical tuning.\n\nWe thank the reviewer for this important point and will include this discussion in the revised manuscript.\n\n> **[Q1]** The paper mentions the existence of more complex paths between probability distributions (e.g., Fisher-Rao or optimal transport geodesics) , whether these more complex geometric structures could lead to superior perturbation strategies? Or why simple linear interpolation proves to be highly effective in mathematical principles?\n> \n\n**[A2]** Thank you for the question. The table below reports PickScore results on 100 prompts from the COCO validation set, comparing linear and Fisher-Rao interpolation paths. (In addition, we initially attempted to implement optimal transport (OT) geodesic interpolation, but found it difficult to implement efficiently)\n\n| *Interpolation Method /* $u$ | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| Linear | 20.65 | 22.15 | 22.32 | 22.32 | 22.21 | 21.90 |\n| Fisher-Rao  | 20.65 | 21.80 | 22.34 | 22.30 | 22.03 | 21.90 |\n\n We observe that Fisher-Rao interpolation provides slightly higher scores at certain $u$ values. However, they also complicate the implementation and may introduce numerical instability.  Given its simplicity, stability, and competitive performance, we adopt linear interpolation in our main experiments. Nonetheless, users may still consider using more complex paths if slightly better quality is desired. We will include these results and discussion in the revised manuscript.\n\n> **[Q2W3]** About the parameter choice of HeadHunter\n> \n\n**[A3]** Thank you for raising this important point. For the number of heads selected per round $k$ (we use $k$ consistently), smaller values allow more accurate assessment of individual additive contributions. We adopt a greedy strategy that selects the head that improves the objective most when added. While $k=1$ provides the most precise search, it increases the number of rounds and total cost. In practice, we found that $k=3$ and $T=5$ offer a good trade-off between efficiency and quality.\n\nThe total number of selected heads is controlled by $k \\times T$. As shown in Fig. 9, the objective typically saturates within a few rounds, and adding too many heads can degrade quality. We empirically observed that selecting around 15–24 heads (approximately the number in a single layer) gives good results.\n\nTo avoid relying on manual tuning of $T$, we also implement an automatic stopping criterion based on the improvement of the objective between rounds. This is a common strategy in deep learning and worked robustly across prompts and models (SD3, FLUX), as shown in Section 4.2 and Appendix D.3.1, supporting its generalizability.\n\n> **[Q2-2]** Are there smarter search strategies (e.g., based on gradients or interpretability scores) that can replace or complement this trial-and-error generation-evaluation cycle?\n> \n\n**[A4]** Thank you for suggesting potential improvements to search efficiency. It is worth noting that exhaustively searching all possible head combinations has exponential complexity $\\mathcal{O}(2^H)$, where $H$ is the total number of attention heads. In contrast, our method already reduces this to $\\mathcal{O}(H)$ by leveraging the compositionality of heads, which represents a significant improvement in efficiency.\n\nTo further improve efficiency, we propose a practical strategy based on our observation of head selection patterns across rounds (Fig. 22, Appendix D.3.2). We found that in early rounds, heads related to structure and general image quality are selected, while in later rounds, heads contributing to style-specific effects are retrieved. This likely occurs because style heads alone do not produce high-quality outputs and thus rely on general-quality heads to be effective.\n\nBased on this insight, once the general-quality head set is obtained (e.g., from a one-time search), we can reuse it as a starting point for style-specific searches. This allows us to skip early rounds and save computational resources up to 40% in our experiments (2 out of 5 rounds). Below, we provide a comparison showing this strategy for the \"sunlit\" and \"spring\" styles:\n\n**Full HeadHunter Search (“sunlit” and “spring” styles):**\n\n| Round | HPSv2 | PickScore | ImageReward | AES | CLIPScore |\n| --- | --- | --- | --- | --- | --- |\n| 0 | 0.2166 | 19.96 | 0.00 | 5.27 | 24.22 |\n| 1 | 0.3094 | 21.66 | 1.48 | 5.77 | 26.35 |\n| 2 | 0.3173 | 22.13 | 1.74 | 5.88 | 27.63 |\n| 3 | 0.3115 | 22.19 | 1.72 | 5.90 | 28.62 |\n| 4 | 0.3133 | 22.09 | 1.78 | 5.92 | 27.95 |\n| 5 | 0.3144 | 22.17 | 1.63 | 5.96 | 28.57 |\n\n**HeadHunter Search Initialized with General-Quality Heads (“sunlit” and “spring” styles):**\n\n| Round | HPSv2 | PickScore | ImageReward | AES | CLIPScore |\n| --- | --- | --- | --- | --- | --- |\n| 2 | 0.3040 | 21.77 | 1.42 | 5.71 | 25.89 |\n| 3 | 0.3102 | 22.01 | 1.67 | 5.98 | 27.06 |\n| 4 | 0.3228 | 22.30 | 1.65 | 5.77 | 27.14 |\n| 5 | 0.3172 | 22.17 | 1.74 | 5.92 | 27.46 |\n\nMore advanced head search methods, such as gradient-based or RL approaches, are promising and will be explored in future work. However, our main contribution lies in showing that attention heads exhibit interpretable, disentangled effects under perturbation, and in proposing a simple yet effective baseline to leverage this. We leave efficiency improvements to future work and thank the reviewer for the valuable suggestion.\n\n> **[Q4]** Users applications that require rapid iteration, this strategy of iteratively generating and evaluating each individual head can become computationally prohibitive. How about the computational efficiency and scalability?\n> \n\n**[A5]** Thank you for the question. Due to space limitations, we kindly refer the reviewer to responses [A1] and [A2] provided to reviewer X5tq, which address this point in detail.\n\n---\n\n[1] Guiding a diffusion model with a bad version of itself, NeurIPS 2024"}, {"review_id": "jkhgy9Kaqk", "reviewer": "Reviewer_xdij", "summary": "This work aims to enhance the performance of image diffusion models in terms of overall fidelity, diversity, prompt alignment, and user-specified criteria through attention perturbation guidance. It builds on two key observations: (1) how to perturb—there is a trade-off in selecting the optimal perturbation strength; and (2) where to perturb—even within the same attention layer, different attention heads vary in their effectiveness for improving specific image generation criteria. To address these challenges, the authors propose SoftPAG and HeadHunter, two methods that improve image generation performance across various metrics by scaling inference-time computation to search for optimal perturbation scales and attention heads. The proposed methods demonstrate effectiveness when applied to SD3 and FLUX models.", "questions": "It is impressive that the FID scores improve monotonically with increased inference-time computation during attention head search, as shown in Figure 7(b). However, a potential concern is that the specific verifier used for guiding the search may be biased toward optimizing a single criterion, potentially at the expense of other important dimensions. To address this, it would be helpful to examine whether the observed performance gains in FID are also reflected in other metrics, such as Inception Score (IS), textual alignment or aesthetic score.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The observation that different MM-DiT attention heads correspond to distinct geometric or appearance-related attributes is both novel and insightful. This could potentially benefit downstream tasks such as zero-shot style mixing and transfer.\n* The paper conducts a thorough analysis of the effects of varying perturbation strengths and attention heads, evaluated across a wide range of metrics, offering a comprehensive understanding of the method’s impact.\n* The proposed method demonstrates strong generalizability across different MM-DiT-based models, showcasing its robustness and adaptability.\n\nWeaknesses:\n* Although searching over attention heads and the interpolation parameter improves performance compared to attention-layer-level search, it introduces significant computational overhead, particularly given the scope of current SOTA image generators (e.g., FLUX1-dev contains 1,368 attention heads). The authors are encouraged to quantify the number of function evaluations (NFEs) and compare them with those required by PAG and SEG, or alternatively, benchmark performance under equal inference-time budgets.\n* In Figure 1&2, a larger $u$ appears to correspond to weaker attention weight perturbations, which is inconsistent with the equations provided in the paper, i.e., $f_I(\\mathbf{A}, u)=u \\mathbf{A}+(1-u) \\mathbf{I}$ and  $f_U(\\mathbf{A}, u)=u \\mathbf{A}+(1-u) \\mathbf{I}$. The authors should clarify this apparent discrepancy.\n* As the core contribution centers on improving image generation quality through increased inference-time computation, it is important to more clearly articulate how this approach relates to, and differs from, existing inference-time scaling methods, particularly the maintream method that involve search over the initial noise space [1].\n\n[1] Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps.", "was_revised": true, "final_justification": "Most of my concerns are addressed. I agree that the generalization ability to new prompts differs the proposed method to previous test-time alignment method. Thus, I raise my score to accept.", "rebuttal": "We sincerely appreciate your time and effort.\n\n> **[W1]** The authors are encouraged to quantify the number of function evaluations (NFEs) and compare them with those required by PAG and SEG, or alternatively, benchmark performance under equal inference-time budgets.\n> \n\n**[A1]** Thank you for raising this important point. We agree that reporting the number of function evaluations (NFEs) is crucial for understanding the efficiency of the proposed method. However, we would like to clarify our interpretation of the question. It is unclear whether the reviewer refers to the NFEs required to generate a single sample, or to the NFEs involved in searching for heads under a specific objective. We interpret it as the latter and respond accordingly. Please kindly correct us if we misunderstood.\n\nThe NFEs for sampling a single image are identical to those of existing layer-level methods employed in PAG and SEG. The difference lies in the **search** phase, where we identify which heads to perturb for a given objective.\n\nAs noted in Appendix D.3.2, we consider following parameters to compute NFE :\n\n- $T$: Total search iteration\n- $M$: Number of prompt-seed pairs\n- $N$: Number of total attention heads in model\n- $S$: Number of denoising steps\n\nIn total, our framework requires $T \\cdot M \\cdot N \\cdot (2S+1)$, considering two forward pass for guidance and one NFE for verifier. In our general quality improvement setting, we use $M = 20$, $T = 1$, and in the style-oriented setting, $M = 5$, $T = 5$. Both use $N = 24 \\times 24$ heads and $S = 20$ steps. On an 8×H100 GPU node, each iteration takes roughly 36 minutes. We emphasize that this one-time cost is amortized due to the strong *generalizability* of the selected heads across new prompts. Please refer to response [A2] to reviewer X5tq for further discussion on generalization to unseen prompts.\n\nRegarding comparison with prior methods such as PAG and SEG, we note that these methods **do not involve any search** procedure. They apply perturbation to pre-selected layers, typically chosen heuristically (e.g., the bottleneck layer of the U-Net). Thus, there is no defined NFEs cost for perturbation **target selection** in those methods, making a direct comparison with HeadHunter infeasible.\n\nIn contrast, one of our core contributions is to **introduce a principled search framework** for selecting perturbation targets based on arbitrary objectives. While we could implement a version of our framework that searches over layers instead of heads, this would still be a comparison between **our own baselines**, not against the original PAG/SEG methods.\n\nWe note that searching over layers would be approximately $H$ times faster per iteration than searching over heads per iteration, where $H$ is the total number of attention heads. However, because layers are a much coarser perturbation unit, the resulting quality is typically suboptimal. We present results comparing layer-level and head-level search below to illustrate this trade-off. For a fair comparison, we set $u = 0.125$ to match the relative influence of the added perturbation unit, as each head-level iteration adds $k = 3$ heads (approximately 1/8 of the heads in a single layer). We also set $T$ to the maximum possible value, equal to the total number of layers.\n\n**Style-Oriented Quality Improvement across Different Perturbation Targets:**\n\n| Search level / metrics | HPSv2 | PickScore | ImageReward | AES | CLIPScore |\n| --- | --- | --- | --- | --- | --- |\n| Layer-level search | 0.2787 | 21.59 | 1.10 | 5.93 | 25.93 |\n| Head-level search | **0.3540** | **24.55** | **1.74** | **6.65** | **32.27** |\n\nThe results show that head-level perturbation guidance outperforms our baseline (PAG with principled search), suggesting that individual heads serve as distinct and fine-grained units contributing to specific visual attributes. We will include these results and the discussion on NFEs in the revised manuscript.\n\n> **[W2]** discrepancy between equations and figures.\n> \n\n**[A2]** We’re sorry for the confusion. We will revise the equations to $f_I(\\mathbf{A},u) = (1-u) \\mathbf{A} + u \\mathbf{I}$ and $f_U(\\mathbf{A},u) = (1-u) \\mathbf{A} + u \\mathbf{U}$ to make it consistent with the figures.\n\n> **[W3]**  it is important to more clearly articulate how this approach relates to, and differs from, existing inference-time scaling methods, particularly the mainstream method that involve search over the initial noise space [1].\n> \n\n**[A3]** Thank you for raising this important point. While both our method and noise-space inference-time scaling share the goal of improving image quality without retraining, there is a key distinction in generalization.\n\nNoise search aims to find an optimal initial noise that yields the best result for a specific prompt under a given objective (e.g., verifier score). However, this search must be repeated from scratch for every new prompt. In contrast, our method identifies a set of perturbation heads that maximize a given objective across a **small set of prompts**, and crucially, this configuration generalizes well to **new prompts**. As shown in Appendix D.3.2 (Tab. 2 and Fig. 26), the retrieved head sets not only improve general image quality for unseen prompts (for general improvement) or for unseen content prompts (for style-oriented improvement).\n\nThis generalization makes our method more practical. For example, model providers or users can share head sets optimized for general quality, preference tuning or specific styles (e.g., \"anime\" or \"line art\"), similar to how LoRA weights are shared in the community. Users can then directly apply these shared configurations without any additional optimization or cost.\n\nMoreover, our method is **orthogonal** to noise search and can be combined with it. Users may first apply HeadHunter to retrieve meaningful heads and then perform inference-time noise search for further improvements.\n\nAlthough our method is optimized using the **average objective across prompts** (to enable generalization), and noise search is optimized **per prompt**, we present a comparison below for completeness. For a fairer comparison, one could also optimize our method on a single prompt, similar to [1], to obtain prompt-specific optimal performance. However, this would deviate from our goal of learning generalizable guidance configurations, so we do not adopt that setting. This comparison uses the **same number of NFEs** for both methods, evaluated under the **style-oriented quality improvement** setting on styles like \"sunlit\" and \"spring\". PickScore is used as the main objective for both.\n\n**HeadHunter:**\n\n| Round | HPSv2 | PickScore | ImageReward | AES | CLIPScore |\n| --- | --- | --- | --- | --- | --- |\n| 0 | 0.2868 | 22.51 | 1.51 | 5.42 | 25.29 |\n| 1 | 0.3309 | 23.59 | 1.83 | 5.98 | 30.38 |\n| 2 | 0.3380 | 24.44 | 1.98 | 6.48 | 30.72 |\n| 3 | 0.3404 | 24.24 | 1.86 | 6.74 | 32.23 |\n| 4 | 0.3587 | 24.70 | 1.99 | 6.61 | 31.43 |\n| 5 | **0.3540** | 24.55 | 1.74 | **6.65** | **32.27** |\n\n**Initial noise search (random search algorithm):**\n\n| Round (same NFEs) | HPSv2 | PickScore | ImageReward | AES | CLIPScore |\n| --- | --- | --- | --- | --- | --- |\n| 0 | - | - | - | - | - |\n| 1 | 0.3444 | 25.56 | 1.98 | 6.38 | 33.32 |\n| 2 | 0.3181 | 25.51 | 1.94 | 6.25 | 30.34 |\n| 3 | 0.3494 | 25.61 | 2.15 | 6.09 | 31.77 |\n| 4 | 0.3541 | 25.41 | 1.84 | 6.25 | 31.59 |\n| 5 | 0.3466 | **25.62** | **2.29** | 6.31 | 31.45 |\n\nWhile noise search achieves higher PickScore on a per-prompt basis, our method achieves **comparable or better performance** in other metrics (e.g., HPSv2, ImageReward, AES, CLIP Score), and crucially, **generalizes to new prompts.** This demonstrates the practicality and robustness of our approach. We will include this comparison and discussion in the revised manuscript.\n\n> **[Q1]** A potential concern is that the specific verifier used for guiding the search may be biased toward optimizing a single criterion, potentially at the expense of other important dimensions. To address this, it would be helpful to examine whether the observed performance gains in FID are also reflected in other metrics, such as Inception Score (IS), textual alignment or aesthetic score.\n> \n\n**[A4]** Thank you for raising this important point. Verifier hacking, or overfitting to a single objective at the expense of other desirable qualities, is a known challenge when optimizing with a single verifier. Previous work on inference-time scaling in noise space has addressed this by using an ensemble of verifiers. Our framework is fully compatible with such ensembles and can easily incorporate multiple objectives if needed.\n\nUnlike gradient-based optimization methods that are often more susceptible to reward hacking, our method does not rely on gradients. Instead, it performs a combinatorial search over the model’s internal structure, which helps it remain more robust against bias in the objective.\n\nImportantly, although we optimize for a single objective such as PickScore, we observe consistent improvements in other metrics, including Inception Score (IS), CLIPScore, and ImageReward. This suggests that our method leads to general image quality improvement rather than overfitting to a specific metric.\n\n**General quality improvement (Fig. 7):**\n\n| metrics / $k$ | 3 | 6 | 9 | 12 | 15 | 18 | 21 | 24 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| FID | 58.79 | 55.02 | 53.04 | 53.08 | 52.52 | **51.34** | 51.90 | 52.52 |\n| IS | 19.48 | 20.47 | 22.14 | 21.62 | 22.69 | 23.40 | 23.93 | **24.09** |\n| PickScore | 20.91 | 21.21 | 21.45 | 21.55 | 21.67 | 21.94 | 22.00 | **22.07** |\n| CLIPScore | 24.40 | 24.78 | 25.03 | 25.08 | 25.15 | 25.29 | 25.26 | **25.32** |\n| ImageReward | -0.09 | 0.11 | 0.26 | 0.30 | 0.33 | 0.51 | 0.54 | **0.59** |\n\n**Style-Oriented Quality Improvement (Sec. 4.2.2):**\nFig. 9 demonstrates that optimizing only PickScore still leads to better performance across other metrics, showing our method’s robustness."}]

Author rebuttal:

Meta-review: ### Summarization.

This paper introduces a generalized framework for attention perturbation guidance in text-to-image diffusion and flow models. The authors make two primary claims: 1) Their framework, which introduces an interpolation parameter between the original and perturbed attention distributions, unifies and improves upon prior methods like PAG and SEG. 2) Specific attention heads correspond to  distinct and interpretable visual attributes (e.g., structure, lighting, texture). Building on this, they propose "HeadHunter," a systematic search method to identify and combine the most effective heads to achieve user-defined objectives, such as enhancing aesthetic quality or reinforcing a specific style, without needing to retrain the model. The findings are validated on large-scale models like Stable Diffusion 3 and FLUX.1, demonstrating improved image quality and style control.

### Strength
- Novel interpretability and controllability: The key insight that individual attention heads correspond to disentangled visual attributes is a significant finding. This enables a new level of fine-grained control over the generation process (acknowledged by all reviewers).
- Thorough experimental analysis: The paper is supported by a comprehensive set of experiments that carefully investigate the effects of different perturbation strategies, layers, and individual heads across a wide range of metrics.(acknowledged by ZmWS, xdij).
- Practicality and scalability: The proposed method is a plug-and-play solution that can be applied to large, pre-trained models without any retraining. The authors demonstrate that the discovered head-sets are generalizable to unseen prompts, making the one-time search cost practical as these configurations can be shared and reused (acknowledged by ZVaq, xdij).
- Principled and automated head selection: The HeadHunter framework provides a systematic, objective-driven approach to designing perturbation strategies, moving beyond the heuristic-based layer selection of previous methods (acknowledged by ZmWS).

### Weakness
- Lack of theoretical depth: Several reviewers (X5tq, ZmWS, ZVaq) pointed out that the paper lacks a deep theoretical motivation for some of its observations. For instance, the reason a "sweet spot" for the interpolation parameter exists and the precise mechanism behind perturbation guidance are explained through intuition and analogy to prior work rather than a formal framework.
- Clarity on hyperparameters and reproducibility: Initially, reviewers (ZmWS, xdij) found it difficult to track the specific hyperparameter values (e.g., guidance scale w and interpolation u) used in different experiments, which could hinder reproducibility. The authors clarified this during the rebuttal.
- Computational cost of headHunter: A primary concern was the computational expense of the HeadHunter search process. Reviewers (X5tq, xdij, ZVaq) questioned its practicality, especially for models with a very large number of heads.

### Reasons for the decision.
I am recommending this paper for acceptance, which is also a consensus among reviewers.  The core contribution, demonstrating that attention heads can be manipulated as interpretable, fine-grained units of control, is a novel and valuable insight for the field of generative models. This finding has the potential to inspire future work in model interpretability, editing, and control. The authors effectively mitigated the weakness during the rebuttal. Two reviewers (X5tq, xdij) raised their scores. There is no negative rating after the rebuttal.

### Summarize the discussion.
The rebuttal period was highly effective and strong. The following points were discussed.

- Computational Cost and Practicality: Reviewers X5tq, xdij, and ZVaq raised major concerns about the practicality of the HeadHunter search due to its high computational cost. The authors clarified the exact number of function evaluations (NFEs), framed it as a one-time cost, and provided new experiments showing that the discovered head-sets generalize effectively to unseen prompts. 
- Comparison to other inference-time methods: Reviewer xdij asked for a clearer distinction from inference-time methods that search over the initial noise space. The authors explained that noise search is per-prompt, whereas their method learns a generalizable configuration. They provided a new experiment with an equal NFE budget, showing their method achieved comparable or better performance on several metrics while having the key advantage of generalization. This successfully addressed the reviewer's concern.
- Theoretical Intuition and Hyperparameter Choices: Reviewers X5tq, ZmWS, and ZVaq requested a better explanation for the method's mechanisms and hyperparameter choices. The authors provided a more detailed intuition by linking perturbation strength  and guidance scale to the direction and magnitude of guidance, drawing an analogy to the Autoguidance framework. They also clarified all hyperparameter settings used in the paper.
- Clarity and Notational Errors: Reviewers xdij and ZmWS noted inconsistencies in the equations for the interpolation parameter.The authors acknowledged the typos and corrected the equations to be consistent with the figures and their implementation.
Decision: Accept (poster)