Title: Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion models

flow matching models

diffusion guidance

generative models
Abstract: Recent guidance methods in diffusion models steer reverse sampling by perturbing the model to construct an implicit weak model and guide generation away from it. Among these approaches, attention perturbation has demonstrated strong empirical performance in unconditional scenarios where classifier-free guidance is not applicable. However, existing attention perturbation methods lack principled approaches for determining where perturbations should be applied, particularly in Diffusion Transformer (DiT) architectures where quality-relevant computations are distributed across layers. In this paper, we investigate the granularity of attention perturbations, ranging from the layer level down to individual attention heads, and discover that specific heads govern distinct visual concepts such as structure, style, and texture quality. Building on this insight, we propose ``HeadHunter", a systematic framework for iteratively selecting attention heads that align with user-centric objectives, enabling fine-grained control over generation quality and visual attributes. In addition, we introduce SoftPAG, which linearly interpolates each selected head’s attention map toward an identity matrix, providing a continuous knob to tune perturbation strength and suppress artifacts. Our approach not only mitigates the oversmoothing issues of existing layer-level perturbation but also enables targeted manipulation of specific visual styles through compositional head selection. We validate our method on modern large-scale DiT-based text-to-image models including Stable Diffusion 3 and FLUX.1, demonstrating superior performance in both general quality enhancement and style-specific guidance. Our work provides the first head-level analysis of attention perturbation in diffusion models, uncovering interpretable specialization within attention layers and enabling practical design of effective perturbation strategies.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to enhance the performance of image diffusion models in terms of overall fidelity, diversity, prompt alignment, and user-specified criteria through attention perturbation guidance. It builds on two key observations: (1) how to perturb—there is a trade-off in selecting the optimal perturbation strength; and (2) where to perturb—even within the same attention layer, different attention heads vary in their effectiveness for improving specific image generation criteria. To address these challenges, the authors propose SoftPAG and HeadHunter, two methods that improve image generation performance across various metrics by scaling inference-time computation to search for optimal perturbation scales and attention heads. The proposed methods demonstrate effectiveness when applied to SD3 and FLUX models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The observation that different MM-DiT attention heads correspond to distinct geometric or appearance-related attributes is both novel and insightful. This could potentially benefit downstream tasks such as zero-shot style mixing and transfer.\n* The paper conducts a thorough analysis of the effects of varying perturbation strengths and attention heads, evaluated across a wide range of metrics, offering a comprehensive understanding of the method’s impact.\n* The proposed method demonstrates strong generalizability across different MM-DiT-based models, showcasing its robustness and adaptability.\n\nWeaknesses:\n* Although searching over attention heads and the interpolation parameter improves performance compared to attention-layer-level search, it introduces significant computational overhead, particularly given the scope of current SOTA image generators (e.g., FLUX1-dev contains 1,368 attention heads). The authors are encouraged to quantify the number of function evaluations (NFEs) and compare them with those required by PAG and SEG, or alternatively, benchmark performance under equal inference-time budgets.\n* In Figure 1&2, a larger $u$ appears to correspond to weaker attention weight perturbations, which is inconsistent with the equations provided in the paper, i.e., $f_I(\\mathbf{A}, u)=u \\mathbf{A}+(1-u) \\mathbf{I}$ and  $f_U(\\mathbf{A}, u)=u \\mathbf{A}+(1-u) \\mathbf{I}$. The authors should clarify this apparent discrepancy.\n* As the core contribution centers on improving image generation quality through increased inference-time computation, it is important to more clearly articulate how this approach relates to, and differs from, existing inference-time scaling methods, particularly the maintream method that involve search over the initial noise space [1].\n\n[1] Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps.\n\nQUESTIONS:\nIt is impressive that the FID scores improve monotonically with increased inference-time computation during attention head search, as shown in Figure 7(b). However, a potential concern is that the specific verifier used for guiding the search may be biased toward optimizing a single criterion, potentially at the expense of other important dimensions. To address this, it would be helpful to examine whether the observed performance gains in FID are also reflected in other metrics, such as Inception Score (IS), textual alignment or aesthetic score.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a method for implementing finer-grained perturbations on modern large-scale text-to-image diffusion and flow models. Existing methods typically perturb at the layer level, this paper targets the perturbation precisely to individual attention heads (HeadHunter). The experiments show that precise control of specific visual attributes (e.g., concept, style, image quality) in the generated image can be achieved by perturbing specific attention heads. This approach overcomes the side effects such as over-smoothing or over-saturation that may result from existing perturbations, and to provide a plug-and-play solution that enhances existing inference pipelines without requiring retraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Experimental designs and analysis are good.\n2. The method can enhances existing inference pipelines without requiring retraining, it will be highly practical and scalable, and can be applied directly to existing pre-trained large diffusion models (e.g., Stable Diffusion 3 and FLUX).\n3. Head-level guidance would allow the user to perform highly precise and targeted manipulation of specific visual attributes of the resulting image.\n\nWeaknesses:\nThe claim made by the work provides a simple approach to finer-graining target style or concept without training and is, in general, innovative. There are sufficient visual results to support the claim. However, the explanations of why some strategies were chosen are relatively weak, with unclear parameter selection. For example, the interpolated parameter u provides intuitive control, but its theoretical optimality is not sufficiently analyzed. A “sweet spot” region is found experimentally, but this is more of an empirical discovery than based on theoretical derivation.\n\n(More specified in the “Question”).\n\nQUESTIONS:\n1. This paper proposes to view attention weights as probability distributions and to perform interpolation, which indeed offers a more unified and intuitive perspective for understanding PAG and SEG. However, this \"unification\" primarily refers to linear interpolation. The paper mentions the existence of more complex paths between probability distributions (e.g., Fisher-Rao or optimal transport geodesics) , whether these more complex geometric structures could lead to superior perturbation strategies? Or why simple linear interpolation proves to be highly effective in mathematical principles?\n2. The basis for the selection of the parameters “number of heads selected per round n” and “maximum number of iterations T” is not specified. How do these parameters affect search efficiency and final results, and are there smarter search strategies (e.g., based on gradients or interpretability scores) that can replace or complement this trial-and-error generation-evaluation cycle?\n3. This paper notes that “adding more perturbed heads results in over perturbation”, which limits the number of heads that can be combined. HeadHunter, while finding good combinations, does not explicitly give a strategy for how to automatically determine the optimal number of combinations (i.e., when to stop adding heads), and still relies on observing performance curves. Does this rely too much on manual evaluation and weaken its generalization?\n4. In the appendix, each concept is associated with different heads of different layers (e.g. Dark and White), and in practice, when users want to optimize certain styles, they need to train to find the corresponding layers and heads first. However, users applications that require rapid iteration, this strategy of iteratively generating and evaluating each individual head can become computationally prohibitive. How about the computational efficiency and scalability?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on current guidance methods for diffusion models, the authors propose a unified attention perturbation guidance model, which interpolates between the original attention map and a perturbed attention map. Specifically, the authors interpret the attention maps as a set of probability distributions, where the uniform distribution represents the scenario with the highest entropy, and the identity matrix can be interpreted as the lowest entropy case. Interpolating between the extremes is investigated experimentally in the first half of the paper, and it is concluded that there exists a certain trade-off. In the second half of the paper, the authors investigate the impact of different layers and attention heads on the guidance methods and propose an iterative framework, called Headhunter, for identifying attention heads that are interpretable and composable.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- Interesting results and thorough investigation\nThe paper goes into great detail investigating the effects of attention-based perturbation guidance on a layer and attention-head level. For the generalized framework of attention perturbation guidance, the authors carefully explore the variety of hyperparameters such as the interpolation coefficient, the guidance strength or the target layers or attention heads. It is interesting to see that selectively guiding certain attention heads impacts specific properties and can be used for controllability purposes.\n\n- Automated framework appears promising\nThe proposed HeadHunter framework allows for a principled and automated approach to find relevant attention heads that improve the generation quality of the image under a predetermined objective function. By composing multiple of these heads, the overall performance can be improved over the unconditional case.\n\n\n### Weaknesses\n- Probability interpretation is mathematically questionable/not well-motivated\nThe interpretation of the attention maps as a set of probability distributions is not well defined in the paper and might be confusing for some readers, as it might imply some stochastic modelling of the attention values. Interpreting the values of the attention map as a set of probability distributions appears to only matter for the argument of minimal/maximal entropy scenarios, which itself does not provide much additional insight into the method. For the minimal entropy scenario, there exists more than just the identity matrix option, which is referred to as 'max guidance' in the Supplementary Material. Technically, the interpolation could be performed on the entire probability simplex, which offers a large number of options.\n\n- Unclear Figures and parameter settings\nThroughout the paper, it remains unclear what value is used for the interpolation parameter u or the guidance scale w in the figures for a lot of scenarios. This makes it hard to understand the exact effects of each parameter and the generalizability of the method.\n\n- Perturbation mechanism remains unclear, mostly empirical observations\nWhile the paper provides a wide variety of different experimental results, it lacks an answer to the deeper mechanisms of the perturbation mechanism. For classifier-free guidance, the intuition behind weighing the conditional signal more strongly stems from sharpening the estimated conditional distribution, but in the case of perturbation guidance, no intuitive interpretation is given.\n\nQUESTIONS:\n- How does perturbation guidance perform compared to regular classifier-free guidance or auto-guidance? \n- How should perturbation guidance be interpreted? How does it affect the estimated probability density path?\n- Are the effects observed for each attention head only valid for attention perturbation, or could it also be used independently? If so, how does it perform compared to the proposed method?\n\nLIMITATIONS:\nYes (in the Supplementary Material)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a unified framework for self-attention score guidance. Basically ablating certain attention head (interpolating between original and identity or flat map) and making them negative control, one can do classifier free guidance with itself and achieve better quality or style. The authors systematically studied the effect of perturbing different attention heads and combination of them. Finally they developed a pipeline to systematically search for heads combinations to optimize certain objective like style.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n- The method is simple to implement and quite principled. The introduction of this method as a conceptual unification of previous guiding method (PAG, SEG) is nice.\n- The systematic study and description of perturbation effect of different head and combination of them is quite interesting (Fig.4,5). It’s thought-provoking to the field for interpretability study in diffusion model.\n- Localizing perturbation to head is interesting and illuminating, from Fig6 seems like it’s a strong and targeted effect.\n\n### Weakness\n- Lack of theoretical framework and further understanding of the observed effect. For example, authors could discuss\n    - it’s a bit unclear if the effect of perturbing a head is equivalent to perturb a layer with less strength $w$. (see questions)\n    - Or perturbing / ablating the attention map in the negative run is equivalent to enhance the map in the positive run.\n- The proposed HeadHunter iterative method is a great proof of concept but it seems a bit *ad hoc,* and expensive… not sure if it’s actually practical. (see questions)\n- Minor: The main text may benefit from a slightly more detailed description of the HeadHunter method, currently all the relevant details are buried in appendix D.\n\nQUESTIONS:\n- What is the complexity of the head hunter method? It seems need to run through all the layer x head and all the prompt once per iteration, that is hundreds of heads and runs. So in a sense it’s very expensive to run this head hunter once. Then the question becomes how useful and generalizable it is? Can the authors show that they can run it once for some prompts and then generalize to many more new prompts / context?\n- As follow up, If so, isn’t the better thing to do is to finetune the model or perform some PEFT specific to those heads? How does the HeadHunter compare to just finetuning?\n- The paper introduced the additional $u$ parameter, but could it be equivalent to tune the cfg guidance scale parameter $w$ for previous methods? \nThis is esp. concerning for the Figure 6 head vs layer comparison, it seems likely that the layer result looks bad because the guidance is too strong….and it’s intuitive that if all attn head in a layer was perturbed, then the diff between two runs will be greater, so the guidance strength should be tune down for it… The authors also noted themselves, at L216, the more head is perturbed, the more likely it is to oversaturate. \n Is there an experiment showing tuning the $u$ parameter is more than tuning guidance scale?\n- In section 3, you introduced an additional interpolation parameter $u$, on the bright side, we have another knob to tune; on the dark side, we have yet another hyperparameter to tune. Is there any optimal choice of this value that could be transferred to new experiments? or does it have high variability from head to head and need to tune separately? Do you need to tune u for each head in HeadHunter?\n- (Minor) L131 Why 2D Gaussian blur cannot be applied to modern MMDiT architecture?\n\nLIMITATIONS:\nYes in appendix H.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}