Title: Entropy Rectifying Guidance for Diffusion and Flow Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative model

diffusion

flows

guidance

sampling

image generation.
Abstract: Guidance techniques are commonly used in diffusion and flow models to improve image quality and input consistency for conditional generative tasks such as class-conditional and text-to-image generation.
In particular, classifier-free guidance (CFG) is the most widely adopted guidance technique.
It results, however, in trade-offs across quality, diversity and consistency: improving some at the expense of others.
While recent work has shown that it is possible to disentangle these factors to some extent, such methods come with an overhead of requiring an additional (weaker) model, or require more forward passes per sampling step. 
In this paper, we propose Entropy Rectifying Guidance (ERG), a simple and effective guidance method based on inference-time changes in the attention mechanism of state-of-the-art diffusion transformer architectures, which allows for simultaneous improvements over image quality, diversity and prompt consistency. 
ERG is more general than CFG and similar guidance techniques, as it extends to unconditional sampling. 
We show that ERG results in significant improvements in various generation tasks such as text-to-image, class-conditional and unconditional image generation.
We also show that ERG can be seamlessly combined with other recent guidance methods such as CADS and APG, further improving  generations.


Peer reviews:
comments: SUMMARY:
This paper proposes Entropy Rectifying Guidance (ERG) which acts as a inference time modification of attention. ERG claims to improve quality and diversity simultaneously. Furthermore the authors also show their method applied to the text encoder and diffusion models separately.

STRENGTHS AND WEAKNESSES:
Weaknesses:
- Not enough transparancy in terms of evaluation. Datasets should be stated more clearly.
- The authors should include ImageNet 256 x 256 using a pretrained DiT-XL/2 and cite the original metrics from the respective papers. 
- Further, a comparison using sFID, InceptionScore, Recall and Precision would be good as this gives even more ground for fair comparisons. 

Strengths:
- Overall well written and interesting concept.
- Metrics support their general claims even though i would like to see a bit more common metrics as means of comparison (see W2/W3)
- Very good performance in the unconditional generation which is interesting.

QUESTIONS:
- Why is ERG so much better in Coverage in the unconditional case while being only marginally better in FID and density?
- Why is there such a sharp change in measured performance in Fig 8?

LIMITATIONS:
The authors discuss limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

comments: SUMMARY:
This paper proposes a novel guidance method to improve sampling quality by updating the model via test-time adaptation. Specifically, during the sampling process, the authors reinterpret the attention mechanism in the self-attention layer as a Hopfield energy formulation and perform updates that descend the Hopfield energy landscape. This approach achieves state-of-the-art results across various evaluation metrics, including FID, recall, coverage, density, prompt fidelity, and VQA scores.

STRENGTHS AND WEAKNESSES:
# Strengths
- The idea of updating attention layers at test time, inspired by hopfield energy is novel.
- The method is simple yet effective.
- The paper is well-written.
- The authors conduct thorough comparisons across many evaluation metrics (FID, recall, coverage, density, prompt fidelity, VQA score).
- Extensive experiments are conducted on various models and datasets.
- The method also performs well in unconditional sampling scenarios.

# Weaknesses & Questions
- In Figure 4, the hyperparameter τ, which is not derived from the Hopfield energy function, appears to be very influential. It would be beneficial to provide a theoretical explanation or motivation for its inclusion.
- Although the method outperforms others on most metrics, it slightly underperforms in FID compared to some baselines. Is there a justification or insight into why this might be the case?

# Minor Weaknesses
- Line 151: Typo. “he state patter” → should be corrected.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Please see weaknesses.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Entropy Rectifying Guidance (ERG), an inference-time guidance method presented as an alternative to classifier-free guidance (CFG) for diffusion/flow-matching models. ERG operates by modifying the energy landscape of attention layers within a diffusion/flow model. Particularly, the softmax temperature in attention is modulated to produce a weaker conditional signal/prediction, which takes the place of the unconditional estimation in CFG. A particularly interesting outcome of ERG is that it can be used to improve unconditional sampling, unlike CFG. The authors provide a set of convincing qualitative and quantitative experiments to demonstrate that ERG improves quality, diversity, and consistency.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nS1. The paper is well written in general. The proposed method, ERG, is well-motivated, aiming to address some of the known shortcomings of CFG.\n\nS2. The proposed method is technically sound, built upon the idea of (a) using a weaker prediction for guidance, and (b) using the hopfield energy formulation of attention to create the weaker prediction.\n\nS3. The authors compare their approach with several baselines for text-to-image generation, class-conditional generation, and unconditional generation. According to the results, ERG seems to simultaneously improve quality, diversity, and prompt consistency, which highlights its potential usefulness.\n\n### Weaknesses\n\nW1. The proposed method seems to be relevant only for a particular family of model architecture (e.g. models with attention layers). Classifier-free guidance is a very general technique that is invariant to the architecture of the generative model.\n\n-  W1.1. Though ERG is theoretically applicable to any diffusion model with attention layers, the experiments seem to have only been performed on DiT-based architectures. It would be more convincing if the paper also covered some UNet architectures that have self-attention (e.g. EDM2).\n\nW2. Classifier-free guidance takes a geometric weighted average between conditional and unconditional probabilities, i.e. $\\hat{p}(x \\mid y) \\propto p(x \\mid y)^{\\omega}p(x)^{1 - \\omega}$. It has a good probabilistic interpretation based on the score function, and can be derived from the expansion of the score function from classifier guidance. It is a bit unclear as to how an unconditional model is being guided by using a weaker unconditional model, as shown in ERG, as both models are unconditional. Is there a probabilistic interpretation of this, beyond \"contrasting with weaker prediction\"?\n\nW3. It is not entirely clear what are the specific novel concepts introduced in this paper over the previous work Smoothed Energy Guidance (SEG). To my understanding, it seems that---(a) modifying the energy landscape of attention layers in diffusion models, and (b) using this phenomenon for performing a sort of guidance in conditional and even unconditional generation---have been previously established by SEG. The primary difference seems to lie in the details of how exactly the energy is modulated (temperature rescaling in ERG vs Gaussian blurring in SEG). Clearly highlighting the differences with SEG will elucidate the strengths of this paper.\n\nQUESTIONS:\nQ1. In section 4.1, the authors mention that they train their own diffusion/flow models for each of the experiments. Can the existing method not be used as a drop-in replacement for CFG in pre-trained diffusion models?\n\n- If training is necessary: why is it required? It seems to me that ERG can be used in pre-trained models, unless I missed something.\n- If training is not necessary: why not provide some experiments with existing open-source pre-trained models?\n\nQ2. Instead of converting some prior work on attention-based UNet architectures to DiT for comparison (SAG, PAG, SEG, noted in L263), it would have been better if the authors could have directly applied their method on the attention-based UNets and compared against the previous methods without modifying them. It will address W1.1. [Unless, ERG requires training a model from scratch (as I noted in Q1), in which case this comparison is not necessary.]\n\nQ3. From my understanding, SEG uses an extra NFE simply because it wants to utilize the already available unconditional pretrained model. Can it not be used similar to ERG in the conditional case without an extra NFE?\n\n- The following ablation would help make things clear: (a) disable temperature scaling in the text encoder (i.e. only using I-ERG), to compare against SEG (w/o extra NFE, if possible). This would prove that the non-local smoothing in ERG (L181) is clearly better than the Gaussian blurring in SEG.\n\nQ4. In L34, the authors note that there is an oversaturation problem in CFG. How does standalone ERG tackle this problem? I did not see any experiments addressing this.\n\n### Suggestions\nS1. I believe there is a small mistake in L304, `At 512 resolution, ERG is best across all metrics, except for coverage where ERG is slightly behind CFG and SEG*`. Referring to Table 3, it seems ERG is better than SEG*, but worse than PAG*. It should probably be `...behind CFG and PAG*` in L304.\n\nIn general, I like the paper, but am concerned with some of the choices in the evaluation procedure (e.g. not using pre-trained model, converting attention UNet-based method to DiT for comparison, etc.).\n\nIf I misunderstood something in the paper or mentioned something that has already been done in the paper, please feel free to clarify my mistake.\n\nI am willing to raise my score by 1 point if some questions are addressed.\n\nLIMITATIONS:\nYes, I believe the authors have addressed their limitations well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Entropy Rectifying Guidance (ERG), a guidance mechanism for diffusion and flow models designed to simultaneously improve the quality, diversity, and input consistency of generated samples, including text-to-image, class-conditional, and unconditional generation tasks. ERG operates by manipulating the attention mechanism, specifically using a Hopfield energy-inspired temperature scaling and attention modification at inference time. This yields a weaker predictive signal contrasted with the original model, thereby providing guidance without the need for an additional model or costly model evaluations. Empirical results demonstrate that ERG outperforms standard Classifier-Free Guidance (CFG) and several recent advanced baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. ERG does not require model retraining, works with pre-trained transformer-based diffusion/flow models, and applies to text-to-image, class-conditional, and unconditional setups.\n2. The approach extends the energy landscape of attention to invent a new, simple guidance mechanism that is broader than prior methods relying on auxiliary or weaker models.\n3. This paper is well-written and well-organized. The proposed method, ERG, is easy to follow.\n\nWeaknesses:\n1. Although ERG, compared to CFG, can be applied to unconditional image generation, the performance improvement, especially in terms of FID scores, is relatively minimal.\n2. The evaluation metrics used in this paper are not very standard. Commonly used metrics, such as Inception Score (IS), sFID, Precision, and Recall, are not reported, which somewhat diminishes the persuasiveness of the experimental results.\n3. Other guidance techniques, such as CFG, can be applied to autoregressive image generation models. Could ERG similarly improve the performance of such models as well?\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}