Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention
Authors: Susung Hong
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Diffusion Guidance
Primary area: diffusion_based_models
TL;DR: This paper proposes Smoothed Energy Guidance (SEG), a novel training- and label-free diffusion guidance method that leverages the energy-based perspective of the self-attention mechanism to enhance image generation.

Abstract: Conditional diffusion models have shown remarkable success in visual content generation, producing high-quality samples across various domains, largely due to classifier-free guidance (CFG). Recent attempts to extend guidance to unconditional models have relied on heuristic techniques, resulting in suboptimal generation quality and unintended effects. In this work, we propose Smoothed Energy Guidance (SEG), a novel training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to enhance image generation. By defining the energy of self-attention, we introduce a method to reduce the curvature of the energy landscape of attention and use the output as the unconditional prediction. Practically, we control the curvature of the energy landscape by adjusting the Gaussian kernel parameter while keeping the guidance scale parameter fixed. Additionally, we present a query blurring method that is equivalent to blurring the entire attention weights without incurring quadratic complexity in the number of tokens. In our experiments, SEG achieves a Pareto improvement in both quality and the reduction of side effects. The code is available at https://github.com/SusungHong/SEG-SDXL.

Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention
Susung Hong" University of Washington
Abstract
Conditional diffusion models have shown remarkable success in visual content generation, producing high-quality samples across various domains, largely due to classifier-free guidance (CFG). Recent attempts extend guidance unconditional models have relied on heuristic techniques, resulting in suboptimal generation qual- ity and unintended effects. In this work, propose Smoothed Energy Guidance (SEG), a novel training- and condition-free approach that leverages the energy- based perspective of the self-attention mechanism to enhance image generation. By defining the energy of self-attention, we introduce method to reduce the curvature of the energy landscape of attention and use the output as the uncondi- tional prediction. Practically, we control the curvature of the energy landscape by adjusting the Gaussian kernel parameter while keeping the guidance scale pa- rameter fixed. Additionally, we present query blurring method that is equivalent to blurring the entire attention weights without incurring quadratic complexity in the number of tokens. In our experiments, SEG achieves Pareto improve- ment in both quality and the reduction of side effects. The code available https: / /gi thub. com catb..coo/Suaaaeeeeeee htt////uuunnooSSSSeseeS SDDSSD
Introduction
Diffusion models [12, 45, 46] have emerged as promising tool for visual content generation, producing high-quality and diverse samples across various domains, including image [38, 40, 42, 8, 13, 30, 2, 24, 9, 29, 34, 33, 4, 41, 5, 20, 22], video [11, 50, 23, 18, 15, 3, 19, 44], and 3D generation [36, 27, 6, 26, 49, 43, 48, 16]. The success of these models can be largely attributed to the use of classifier-free guidance (CFG) [14], which enables sampling from a sharper distribution, resulting in improved sample quality. However, CFG is not applicable to unconditional image generation, where no specific conditions are provided, creating disparity between the capabilities of text-conditioned sampling and sampling without text. This disparity results in restriction in application, e.g., synthesizing images with ControlNet[51) without a text prompt (see the last two columns of Fig. 1).
Recent literature [17, 1] has attempted to decouple CFG and image quality by extending guidance to general diffusion models, leveraging their inherent representations [25, 32, 17]. Self-attention guidance (SAG) [17] proposes leveraging the intermediate self-attention map diffusion models to blur the input pixels and provide guidance, while perturbed attention guidance (PAG) [1] perturbs the attention map itself by replacing with an identity attention map. Despite these efforts, these methods rely on heuristics to make perturbed predictions, resulting in unintended effects such as smoothed-out details, saturation, color shifts, and significant changes in the image structure when given a large guidance scale. Notably, the mathematical underpinnings of these unconditional guidance approaches are not well elucidated.
"This work was mostly done at Korea University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"a Bonsai tree growing vintage typewniter, minimolist style"
"o blue jay standing on large roinbow macorons"
(a) Vanilla SDXL
(b) SDXL Smoothed Energy Guldance (Ours)
Figure 1: Teaser. (a) Images sampled from vanilla SDXL [35] without any guidance. (b) Images sampled with Smoothed Energy Guidance (Ours). Ø denotes that there is no condition given. With various input conditions, and even without any, SEG supports the diffusion model in generating plausible and high-quality images without any training.
In this work, we approach the objective from energy-based perspective of the self-attention mechanism, which has been previously explored based on its close connection to the Hopfield energy [39, 31, 7]. Specifically, start from the definition of the energy of self-attention, where performing a self-attention operation equivalent to taking a gradient step. light this, we propose tuning- and condition-free method that reduces the curvature of the underlying energy function by directly blurring the attention weights, and then leverages the output as the negative prediction. We call this method Smoothed Energy Guidance (SEG).
SEG does not merely rely on the guidance scale parameter that cause side effects when its value becomes large. Instead, we can continuously control the original and maximally attenuated curvature of the energy landscape behind the self-attention by simply adjusting the parameter of the Gaussian kernel, with the guidance scale parameter fixed. Additionally, we introduce a novel query blurring technique, which is equivalent to blurring the entire attention weights without incurring quadratic cost in the number of tokens.
We validate the effectiveness SEG throughout the various experiments without and with text conditions, and ControlNet [51] trained on canny and depth maps. Based on the attention modulation, SEG results in less structural change from the original prediction compared to previous approaches [17, 1], while achieving better sample quality.
2 Preliminaries
2.1
Diffusion models
Diffusion models [12, 45, 46] are class of generative models that generate data through an iterative denoising process. The process of adding noise to an image x over time [0, T is governed by the forward stochastic differential equation (SDE):
dx = f(x, t)dt + g(
(1)
where and g are predefined functions that determine the manner in which the noise is added, and dw denotes a standard Wiener process.
Correspondingly, the denoising process can be described by the reverse SDE:
dx [f(x t) 2 V log P: (x)]dt g(t)dw
(2)

---

where Vx log (x) represents the score of the noisy data distribution and dw denotes the standard Wiener process for the reversed time.
Diffusion models are trained to approximate the score function with so 5o(x, L) & Vx P p (x). To condition e.g., class label text, one simply needs train diffusion
generate image based models to approximate the conditional score function with Sp(x, f, c) R Vx log Pt (x|e) and replace Vx log Pr (x) with it in the denoising process. To enhance the quality and faithfulness of the generated samples, classifier-free guidance (CFG) [14] is widely adopted. Accordingly, the reverse process becomes:
dx [f(x,t) g(t) (Tiggoo (xx,,, (, (Tay 1)so (x, t)) g(t)dw.
Here, (x, () is learned by dropping the label by certain proportion, and Jofg is hyperparameter that controls the strength of the guidance. Intuitively, CFG helps us to sample from sharper distribution by conditioning on class label or text.
2.2 Energy-based view of attention mechanism
The attention mechanism [47], which has been widely adopted in diffusion models [12], has been interpreted through the lens of energy-based models (EBMs) [3,,39, 7]. especially through its close connection with the Hopfield energy [7, 39]. In the modern (continuous) Hopfield network, the attention operation can be derived based on the concave-convex procedure (CCCP) from the following energy function [39]:
E(E) = -se(
+
(4)
2
where & E Rixd X E RNdd and lse stands for the log-sum-exp function, defined as Ise(v) := The quadratic term acts regularizer prevent & from exploding [39], while - -se( penalizes misalignment between X and &.
N log E e":)
Mathematically, it turns that the attention mechanism is equivalent to the update rule of the modern Hopfield network [7, 39]. Specifically, inspired by the Hopfield energy in (4), and noticing that the first term depends on the attention weights, we propose the following energy function for entire self-attention weights in diffusion models:
Definition 2.1 (Energy Function for Self- -aff-nttention. Let Q E (Hd))xd be matrix of query vectors and E (H be a matrix of key vectors, where H, W, and d represent the height, width, and dimension, respectively. Let A E (((H))(HW)) == QK The energy function with respect to entire
self-attention weights in diffusion models is defined as:
W
H W ΣΣ
E(A) X>''EEE El(as)))
E'(a) == -sse (a) - log
(5)
Note that to explicitly denote the spatial dimension, we use the subseript (æ,, y) to represent the index of row column of the matrices. Despite using the definition in (5) for the rest of the paper for simplicity, we additionally discuss the dual case, where we use the swapped indexing, in Appendix B.
This view leads us important intuition: the attention operation can be seen minimization step on the energy landscape, considering that the first derivative represents the softmax operation which also appears in the attention operation. Building upon this intuition, we argue that Gaussian blurring on the attention weights modulates the underlying landscape to have less curvature, and we demonstrate this in the following sections by analyzing the second derivatives.
3 Method
Our aim is to theoretically derive the effect of Gaussian blur applied on the attention weights, which in the end attenuates the curvature of the underlying energy function. Then, utilizing this fact, we develop attention-based drop-in diffusion guidance that enhances the quality of the generated samples, regardless of whether an explicit condition is given. In Section 3.1, we claim some useful properties of Gaussian blur: that it preserves mean, reduces variance, and thus decreases the Ise value. In
3

---

Section 3.2, we find that the curvature of the energy landscape is attenuated by the attention blur operation, leading naturally to blunter prediction for guidance. And finally, in Section 3.3, built upon this fact, we define Smoothed Energy Guidance (SEG) and propose the equivalent query blurring method, which can perform attention blurring while avoiding quadratie complexity in the number of tokens.
3.1 Gaussian blur to attention weights
In this section, derive some important properties of the Gaussian blur with the aim of figuring the variation of the energy landscape. To this end, we start from some mathematical underpinnings on applying Gaussian blur attention weights.
Gaussian filter is a convolution kernel that uses 2D Gaussian function to assign weights to neighboring pixels. The 2D Gaussian function is defined as:
(=-pp)"+(x-p)))
G( y) 2TTO2
20
where pe and My are the means in the x and y directions, and a is the standard deviation. The 2D Gaussian filter possesses symmetry, i.e., G(x.y) G(-z, -y), and normalization, i.e., IJ G(z, y)drdy = 1. In practice, we use a discretized version of the Gaussian filter with a fi- nite kernel size depending
a, normalized to sum 1.
Lemma 3.1. Spatially applying a 2D Gaussian blur to the attention weights a == Qk' preserves the average Bi (1,)) In addition, the variance monolonicully decreases every time apply the Gaussian blur.
Proof sketch. Applying a 2D Gaussian filter to the attention weights ((i.) yields the blurred values ((s.)):
k
k
a(i,5) = Σ
G(m,n) ((itm.jtn)
ITL - Kn=
where le is the filter size, G(m, n)) is the Gaussian filter value at position (m, and a(i+mm. is the attention weight at position (i m. n). Since the Gaussian filter symmetric and normalized. it can be shown that the mean of the blurred attention weights is equal to the mean of the original attention weights. Similarly, we can show that the variance monotonically decreases when we apply a 2D Gaussian filter. See Appendix A. for the complete proof.
Note that this fact also implies that blurring with Gaussian filter with larger standard deviation causes a greater decrease in the variance of attention weights. This is because a Gaussian filter with larger standard deviation can always be represented as convolution of two filters with smaller standard deviations, due to the associativity of the convolution operation.
Finally, show that applying 2D Gaussian blur to attention weights increases the Ise value in (5), i.e., increases the energy in (5). This provides a bit of intuition about the underlying energy landscape, yet it is more prominently utilized in the claims in the following sections.
Lemma 3.2. Applying a 2D Gaussian blur to attention weights a == Qk" increuses the Ise term when we consider the second-order Taylor series approximation of the exponential function functiio around the mean μ = [E.](a(s.)) Consequently, the maximum is achieved when the attention is uniform, i.e., ((.)) ((k.I) Vi,J, k, This corresponds to the case when we apply the Gaussian blur with a co.
Proof sketch. Applying the second-order Taylor series approximation around the mean ja, and using Proposition 3.1, show that the second-order approximation of Ise(a) is larger than equal that of lse(ā). Subsequently, we introduce Lagrange multipliers to find the maximum, which gives us the result, ((.))) = a(e.) Vi,j, k. We leave the full proof in Appendix A.2.
3.2 Analysis of the energy landscape
In this section, we demonstrate that applying a 2D Gaussian blur to the attention weights before the softmax operation results in computing the updated value with reduced curvature of the underlying energy function. To this end, we analyze the Gaussian eurvature before and after blurring the attention weights. This is closely related to the Hessian of the energy function.

---

Theorem 3.1. Let the attention weights be defined as a= (5). Then, applying Gaussian blur to the attention weights a before the softmax operation results in the attenuation of the Gaussian curvature of the underlying energy function where gradient descent performed.
Qk". Consider the energy function in
Proof sketch. Let H denote the Hessian of the original energy function, i.e., the derivative of the negative softmax, and H denote the Hessian of the new energy function associated with blurred attention weights. Furthermore, let by denote the i-th row, j-th column entry in the Toeplitz matrix B representing the Gaussian blur. Calculating the derivatives, we have the elements of the Hessians, hi (E(a); (a)) and hig (E(ā); 6ig :b) Using Lemmas 3.1 and 3.2 and under reasonable assumptions, we observe that (H) (e)(H)), which implies that the minimization step is performed on a smoother energy landscape with attenuated Gaussian curvature. The full proof Appendix A.3.
To provide more intuition about what is actually happening and how we utilize this property in the later section, it is intriguing consider the attenuating effect the curvature in analogy to classifier-free guidance (CFG). CFG uses the difference hetween the prediction based on the sharper conditional distribution and the prediction based the smoother unconditional distribution to guide the sampling process. By analogy, we propose method to make the landscape of the energy function smoother guide the sampling process, opposed the original (sharper) energy landscape.
From a probabilistic perspective, the energy is associated with the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood as shown Lemma 3.2, and also reduces the curvature of the distribution shown in Theorem 3.1.
3.3 Smoothed energy guidance for diffusion models
Based on the above observation that the Gaussian blur on attention weights attenuates the curvature of the energy function, propose Smoothed Energy Guidance (SEG) in this section. For brevity, redefine the unconditional score prediction as (x. t), and the unconditional score prediction with the energy curvature reduced as (x, t). Specifically, $o(x, i) is the prediction with the attention weights blurred using a 2D Gaussian filter G with the standard deviation a. We formulate the process
(QK )seg = G * (QK)
(6)
where * denotes the 2D convolution operator. Then, we replace the original attention weights with (QK sog, and compute the final value as in ordinary self-attention.
For practical purposes when the number of tokens is large, propose efficient computation of (6) using the property a linear map, since the convolution operation is linear. Concretely, blurring queries is exactly the same as blurring the entire attention weights, and we propose the following proposition to justify our claim.
Proposition 3.1. Let Q and K be the query and key matrices self-attention, and let G be a Gaussian filter. Blurring the attention weights with G is equivalent to blurring the query matrix Q with G and then computing the attention weights.
Proof. Since the convolution operation is linear, always find Toeplitz matrix B such that:
(QK = B(QKT),
(7)
where denotes the 2D convolution operation. Using the properties of matrix multiplication, we can rewrite (7) as:
B(QK [ = (G Q
(8)
Finally, SEG is formulated as follows:
dx [f(x, t) (x (Yoog (x, g(t)dw,
(9)
where Yeeg denotes the guidance scale of SEG.
5

---

(SDXL)
0=1
o=2
10
Figure 2: Unconditional generation using SEG.
0 (SDXL)
==
0 o=2
0
==
10
00
"a Jellyfish playing the drums in an underwater concert"
"a high-resolutioon satellite image of a bustling shipping port, countless colorful containers"
Figure 3: Text-conditional generation using SEG.
In straightforward manner, as SEG does not rely on external conditions, it can be used for conditional sampling strategies such as CFG [14] and ControlNet [51]. For the combinatorial sampling with CFG, following [17], we simply extend (9) for improved conditional sampling with both SEG and CFG as follows:
dx = [f(x,t) t) - ((1))))((((((1()))))))) + "Sseg)s((x t) + SetgSe f, Sp t)) + g())dw (10)
which an intuitive result, as the update rule moves x towards the conditional prediction while keeping it far from the prediction with blurred attention weights.
We are likely to get result with saturation when using large guidance scale, such as with classifier- free guidance (CFG) [14], self-attention guidance (SAG) [17], and perturbed attention guidance (PAG) [1]. This is significant caveat since need increase the scale achieve maximum effect with these methods. Contrary to this, we can fix the scale of SEG as justified Sec. 5.5 and control its maximum effect through a of the Gaussian blur, making the choice more flexible. For a, two extreme cases are recognized. If a + 0, the blurred attention weights remain the same as the original, while when + oc, the attention weights merely adopt single mean value across spatial axes. We find that even the latter extreme case results in high-qualig outcome, corroborating that we can control the quality to the limit without saturation.
4 Discussion on related work
Classifier-free guidance (CFG) [14], first proposed as replacement for classifier guidance (CG) [8] controlled by scale parameter. The higher we set classifier-fres guidance, the more we get faithful, high-quality images. However, it requires externa labels, such text [30] class [8] labels, making impossible to apply to unconditional diffusion models. Also, requires specific traning procedure with label dropping and it is known that high CFG causes saturation [42].

---

Condition 0 (SDXL) o=1 o=2 0=5
10
Figure 4: Conditional generation using ControlNet [51] and SEG.
Table 1: Quantitative comparison of SEG with vanilla SDXL [35], SAG [17], and PAG [1] for
unconditional generation.
SEG
SEG
Metric
Vanilla SDXL [35]
SAG [17]
PAG [1]
10
FID
129.496
106.683
105.271
95.316
88.215
LPIPS
0.706
0.542
0.522
0.536
LPIPS
0.644
0.472
0.454
0.472
Tackling the caveats of CFG, unconditional approaches such as self-attention guidance (SAG) [17] and perturbed attention guidance (PAG) [1] have been proposed. SAG selectively blurs images with the mask obtained from the attention map and guides the generation process given the prediction. This indirect approach causes saturation and noisy images when given large guidance scale, leading to the selection of a guidance scale less than or equal to 1. PAG guides images using prediction with identity attention, where the attention map an identity matrix. However, the reliance on heuristics to make perturbed predictions results in unintended side effects. As an example of the side effects of replacing the attention map with identity attention. PAG changes the visual structure and color distribution of an image, as evidenced in Figs. 5, 8, and 9.
Contrary to these, we control the effect of SEG through the standard deviation of the Gaussian filter, a. Moreover, while being theory-inspinn SEG is relatively free from unintended effects. In the following section, corroborate our claim with extensive experiments.
5 Experiments
5.1
Implementation details
We build upon the current open-source state-of-the-art diffusion model, Stable Diffusion XL (SDXL) [35], as our baseline, and do not change the configuration. To sample with SEG, we choose the same attention layers (mid-blocks) and guidance scale as PAG [1]. For SEG and PAG sampling, we use the Euler discrete scheduler [21], while for SAG [17], we instead use the DDIM scheduler [45] since the current implementation of SAG does not support the Fuler discrete sampler. For SAG and PAG, we use the same configuration they used in the experiments with the previous version of Stable Diffusion, with guidance scales of and 3.0, respectively. We set %ey 3.0, except in the ablation study.

---

Vanilla SDXL
SAG
PAG
SEG (a 10)
SEG (a co)
Figure 5: Qualitative comparison of SEG with vanilla SDXL [35], SAG [17]. and PAG [1].
Table 2: Text-conditional sampling with different
SEG
Metric
Vanilla SDXL [35]
10
FIDL
53.423
48.284
41.784
33.819
29.325
26.169
CLIP Scoret
0.271
0.273
0.278
0.285
0.290
0.292
LPIPS gg
0.361
0.410
0.449
0.472
0.493
LPIPS alex
0.295
0.347
0.390
0.416
0.440
5.2 Metrics
We use various metrics to evaluate quality (FID [10] and CLIP score [37], calculated with 30k references from the MS-COCO 2014 validation set [28]) and to assess the extent of change due to applied guidance (LPIPS alex [52]). The latter metric, calculated using the outputs of vanilla SDXL, measures the extent of side effects by comparing guided images to their unguided counterparts.
5.3 Controlling image generation with the standard deviation
In this section, aim is to demonstrate that with SEG, can sample plausible images using vanilla SDXL [35] under various conditions and even without any conditions, as demonstrated in Fig. 1. Furthermore, without the risk of saturation, we control the quality and plausibility of the samples. For the results, we use {1. 2, 5, 10}. Additionally. as mentioned in Sec. 3.3, we present two extreme cases, σ (vanilla SDXL) and + co (uniform queries).
Unconditional generation In this section, our aim is to demonstrate that with SEG, we can sample plausible images from the unconditional mode of the vanilla SDXL, which was originally trained on large-scale text-to-image dataset. The results are presented in Fig. 1, Fig. 2, and Table 1. The results show clear tendency draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms.

---

In Fig. 2 and Fig. 13, we show the effectiveness of generating more plausible images, while vanilla SDXL unable to generate high-quality images without any conditions. The results show a clear ten- dency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms. When a is larger, the deñniitin and expression of the samples improve, as the difference in curvature becomes more pronounced.
Conditional generation Figs. 3, 4, 10, 11, and 14, we display sampling results conditioned on text, Canny, and depth map. Using text (Fig. 3), the vanilla SDXL without CFG is unable to generate high-quality images and produces noisy results. Canny and depth map conditioning on SDXL (Fig. 4, 10, and 11) is achieved through ControlNet [51], trained on such maps. The results show that SEG enhances the quality and fidelity of the generated images while preserving the textual and structural information provided by the conditioning inputs. Notably, increases, the generated images exhibit improved definition and quality without introducing sigmificant artifacts or deviations from the original condition. The combination with higher CFG scales is shown in Figs. 15-19.
In Table 2, we show the quantitative results for text-conditional generation in terms of a. We observe a clear trade-off between image quality (represented by FID and CLIP score) and the deviation from the original sample (represented by LPIPS). We sample 30k images for each a to compute the metrics.
5.4 Comparison with previous methods
Since the results are visually favorable when we use o 10 and σ - 0o, and they are the best in terms of CLIP score and FID, respectively, we adopt those configurations for comparison of unconditional guidance methods. The results are presented in Figs. 5, 8, 9, and Table Notably, our method achieves better image quality in terms of FID, while remaining similar to the original output of vanilla SDXL as measured by LPIPS, implying Pareto improvement.
5.5 Ablation study
FID Scores
CLIP Scores
In this section, we address two parameters, Tsg and a, and justify that fixing 'syy is a reasonable choice. In Fig. 6, we present the results from our testing. The results reveal that increasing "Seg does not gen- erally lead to improved sample quality in terms of FID and CLIP score, due to various issues such as saturation. In contrast, increasing a tends to im- prove sample quality and plausibility. This supports the claim that image quality should be controlled by o, instead of the guidance scale parameter. We sam- ple 30k images for each combination to calculate the metrics.
Yag
55
=3
Fige 5
50-
45
CUP Score
40
0.275
Vice m
5
0.270
10
10
Figure 6: Ablation study on Yso and
6 Conclusion, limitations and societal impacts
Conclusion We introduce Smoothed Energy Guidance (SEG), novel training- and condition-free guidance method for image generation with diffusion models. The key advantages of SEG lie in its flexibility and the theoretical foundation, allowing us to significantly enhance sample quality without side effects by adjusting the standard deviation of the Gaussian filter. We hope our method inspires further research on improving generative models, and extending the approach beyond image generation, for example, to video or natural language processing.
Limitations and societal impacts The paper proposes guidance to enhance quality outcomes. Consequently, the attainable quality of our approach is contingent upon the baseline model employed. Furthermore, the application of SEG to temporal attention mechanisms in video or multi-view diffusion models is not addressed, remaining a promising avenue for future research. It is important to note that the improvements achieved through this method may potentially lead to unintended negative socictal consequences by inadvertently amplifying existing stereotypes or harmful biases.

---

Acknowledgements
I would like to express my gratitude Yong-Hyun Park, Junha Hyung, and Donghoon Ahn for their valuable feedback and insights. Their thoughtful comments and suggestions have been instrumental in improving this work.
References
[1] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim. SeonIww Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectffying diffusion sampling with perturbed-atiente guidance. arXiv preprint arXiv:2403.172 173777 2024.
[2] Georgios Batzolis, Jan Stanezuk, Carola- Bibiane Schönlich, and Christian Etmann. Conditional image generation with score-based diffuusion models. arXiv preprint arXiv:2111. 2021.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[4] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval-augmentee diffusion models. Advances in Neural Information Processing Systems, 35:15309-15324, 2022.
[5] Manuel Brack, Felix Friedrich, Katharia Kormmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8861-8870, 2024.
[6] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appcarance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246-22256, 2023.
[7] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-299 299, 2017.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 14:8780-8794, 2021.
[9] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances Neural Information Processing Systems, 36:16222 -6112222-16239, 2023.
[10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300 2022.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 6851, 2020.
[13] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2007.12598. 2022.
[15] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633-8666 2022.
[16] Susung Hong. Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 24 diffusion for view-consistent text-to-3d generation. Advances in Neural Information Processing Systems, 36, 2024.
[17] Susung Hong. Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusson models using self attentotnn guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7462 1471, 2023.
10

---

[18] Susung Hong, Junyoung Seo, Heeseong Shin, Sunghwan Hong, and Seungryong Kim. Direct2v: Large language models are frame-level directors for zero-shot text-to- video generation. arXiv preprint arXiv:2305.14330 14330, 2023.
[19] Hanzhuo Huang. Yufan Feng. Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator. Advances in Neural Information Processing Systems, 36, 2024.
[20] Ziqi Huang, Kelvin CK Chan, Yuming Jiang, and Ziwei Liu. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6080-6090, 2023.
[21] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[22] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[23] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15964, 2023.
[24] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2426-2435, 2022.
[25] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 10960, 2022.
[26] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffussin for consistent text-to-3d, arXiv preprint arXiv:3310.02596, 2023.
[27]
Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to- 3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-30% 2023.
[28] Tsung-Yr Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[29] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua Tenenhaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439. Springer, 2022.
[30] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
[31] Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye, Energy-based cross attention for bayesian context update in text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[32] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information P Processing Systems, 36:24129 2224129-24122 2023.
[33] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
[34] Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image generators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10199- 10208, 2023.
11

---

[35] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01990 2023.
[36] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI.R, 2021.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar arXiv:220.00612,, 1(2)3, 2022.
[39] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008. 2020.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11684-10655, 2022.
[41] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Flect, and Mohammad Norouzi. Palette: Imagc-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36494, 2022.
[43] Junyoung Seo, Susung Hong, Wooseok Jang, Inès Hyeonsu Kim, Minseop Kwak, Doyup Lee, and Seungry- ong Kim. Retrieval-augmented score distillation for text-to-3d generation. arXiv preprint arXiv:2402.02972 2024.
[44] Uriel Singer. Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint Xiv: 14792, 2022.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint axXwv20110002502, 2020.
[46] Yang Song, Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rX1v:201..13456, 2020.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[48] Haochen Wang, Xiaodan Du Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12619- 12629, 2023.
[49] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[50] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwon Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18456-18466, 2023.
[51] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3847, 2023.
[52] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586 595,5995.220 2018.
12

---

A Full proofs
A.1 Proof of Lemma 3.1
Let ((t,)) denote the original attention weights and ((s,)) denote the blurred attention weights, as in the main paper. Assume that the original attention weights properly padded to maintain consistent statistics. Then, the following shows that the mean of the blurred attention weights remains the same.
B (((t,)))
((i)))
Σ
G(m, a(t+m.f+x))
HW
HW
im
k
k
H
w ((+mm++))
= Σ
G(m,n). n)
HW
m=
k n=-k
5=1
k
n) Eesl( = Eigla
Σ
G(m, n) = E., ((.))
In addition, the variance of the blurred attention weights smaller than or equal to the variance of the original attention weights.
W
Var. (a(
(6.5)
(.,l)(,))))
HW
H
W
2
G(m, n)) (a(tmmttt) [(
HW
i=1
m
k
kn=
k
Σ
Σ
Σ
n) Girr,) s) Cov(a(itm.jtnn) a( ((-rr+t))
M= d -== kro
Using the CCuuchySShwaar inequality and the normalization property of the 2D Gaussian filter, can show that the variance monotonically decreases when we apply Gaussian blur.
&
Var., (ar.) S
Σ Σ Σ
G(m,n) G(r, s) V Varla Varla(oera+s))
m= k n==
k
2
G(m, n)
VVr(autmmsnn)
M=
R==
k Σ
G(m, m)
Σ G(m,n) Varlarotssta))
m=
k n
k
G(m,n) Var(a(tmsstn)
kn=-k
(ara)(a(ss.))
A.2 Proof of Lemma 3.2
Applying the second-order Taylor series approximation of et to our function f around the mean μ. get:
#
W
H w
el (" - 1)+ _ (((())))) - -13²)
R
(11)
1=1
1=1
w
HW
el + el
(a(s.) - 12)
(12)
13

---

In the last step, we used the fact that Σ (==1(((i)) - ()) Similarly,
because μ is the mean.
H w
ΣΣ (ua) R HW et + e
(a(5))) 1))
(13)
Since Var[a] > Var[a]. we have:
" w
11 W
Σ ΣΣ (a())~)))
-1)²
Σ Σ (a p.)
(14)
2= i=1.=1
Therefore, the second-order approximation of Ise(a) is larger than that of Iso(a).
Note that this fact also implies blurring with Gaussian filter with bigger variance causes more decrease in the variance of attention weights, because Gaussian filter with larger variance can always be represented as a convolution of two filters with smaller variances, and the convolution operation associative.
To find the maximum value subject to the constraint ((1,1) + ((12)) +... + ((H.W) = c for some constant c, we introduce Lagrange multipliers. Let (((().)), (.)) ((H,w) = ((1,)) + ((12) +
... ((H,W). The Lagrangian function is defined as:
L(Q(L.)) ((1,2) ((H,W): X) = e te + t.. - (Q (.( -c)
(15)
Taking partial derivatives and setting them to zero yields:
(lii)) - ==
(16)
Oa(.,)
Solving ((t.), we obtain ((s,)) In(A) for all i = 1,2,.... and = 1,2,..., W Summing these equations results in:
À = e HW
(17)
Substituting back into a(i.)) a(i-))= In()) gives ((1.1) = ((1.2) = = ((H,W) HWW Therefore, the minimum value of Σ W (a(aa) is achieved when a(1.)) a(1.)) = =... -(H,W)-
A.3 Proof of Theorem 3.1
Let a = (a)...., an) denote the attention values before the softmax operation, and let å (ā1,...,ū, an) denote the attention values after applying the 2D Gaussian blur. Let H denote the Hessian of the original energy, i.e., the derivative of the negative softmax, and H denote the Hessian of the underlying energy associated with the blurred weights.
The elements in the i-th row and j-th column of the Hessian matrices given by:
huj (E(a), (a)),
(18)
hig = (E(a); (
(19)
respectively, where by are the elements of the Toeplitz matrix corresponding the Gaussian blur kernel, and dig denotes the Kronecker delta.
Assuming (ā)) x 0 and E(a),{(a); 0 for all and j. which is reasonable assumption when the number of token large and the softmax values get small, the non-diagonal elements the Hessians approximate to 0 and the diagonal elements dominate. Therefore, the determinants of the Hessian matrices are approximated as the product the dominant terms:
det (H)
II ((a):: det
II ba)
(20)
14

---

We have the following inequality:
e R3
I ((a),
{(a);
e
(21)
T1 jmnl
e%s ))
- 4z
II ((a)
(22)
2 ens
where the first inequality follows from the property of the Gaussian blur kernel, 0 S by < 1, and the second inequality is derived from Lemmas 3.1 and 3.2, which demonstrate the mean-preserving property and the decrease in the lse value when applying a blur. The monotonicity of the logarithm function implies that the denominator involving the blurred attention weights is smaller. Eventually, we obtain the following inequality:
(det(H)) < (H))
(23)
This implies that the updated value is derived with attenuated curvature of the energy function underlying the blurred softmax operation compared to that of the original softmax operation. n
B Dual definition
As we previously stated in Section 2.2, we have the dual definition regarding (5), where we use swapped indexing. Importantly, the swapped indices can be interpreted as altering the definition of attention weights to A : KQ
similar conclusion can be drawn as the main paper, except that query blurring becomes key blurring with this definition. To see this, Eq. 7 changes slightly with this definition, using the symmetry of the Toeplitz matrix B:
G (KQ*) B(KQ ((KQ)) (TQQ))B!
(24)
T
(25)
-QKTB TTT
(26)
-(Q(KK))) TTT (Q(G++))T) =G+*K)Q
(27)
(28)
(29)
where denotes the 2D convolution operation. Empirically, this altered definition does not introduce significant difference in the overall image quality, shown in Fig. 12.
C Additional qualitative results
In this section, we present further qualitative results demonstrate the effectiveness and versatility of our Smoothed Energy Guidance (SEG) method across various generation tasks and comparison with other approaches.
Comparison with previous methods Figs. 8 and 9 provide qualitative comparison of SEG against vanilla SDXL [35], Self-Autentin Guidance (SAG) [17], and Perturbed Attention Guidance (PAG) [1]. These comparisons highlight the superior performance of SEG in terms of image quality, coherence, and adherence to the given prompts. SEG consistently produces sharper details, more realistic textures, and better overall composition compared to the other methods.
Conditional generation with ControlNet Figs. 10 and 11 showcase the application of SEG in conjunction with ControlNet [51] for conditional image generation. These results illustrate how SEG can enhance the quality and coherence of generated images while maintaining fidelity to the provided control signals. The images demonstrate improved detail, texture, and overall visual appeal compared to standard ControlNet outputs without prompts.
15

---

Attention
Attontion
Attention Weights
Energy Landscape
Attention Weights Energy Landscape
(p((
K
" K
(x.
Apply E (0)
[a]
(b)
[c]
Figure 7: Pipeline of SEG. (a) Original sampling process, self-attention weights, and the correspond- ing energy landscape. (b) Our modified sampling process with blurred queries where a E (0, oc), inducing blurred attention weights and the corresponding smoothed energy landscape. (c) A concep- tual figure of "seg. Note that since the guidance linearly extrapolates predictions from (a) and (b), high guidance scale causes samples to be out the manifold.
Unconditional and text-conditional generation Fig. 13 demonstrates the capability of SEG in unconditional image generation, showcasing its ability to produce high-quality, diverse images without text prompts. Fig. 14 exhibits text-conditional generation results using SEG, illustrating its effectiveness in translating textual descriptions into visually appealing and accurate images.
Interaction with classifier-free guidance Figs. 15-19 present a series of experiments exploring the combination of SEG with CFG. In these experiments, the SEG guidance scale (Tseg) is fixed at 3.0, while the CFG scale is varied. The results demonstrate that SEG consistently improves image quality across different CFG scales without causing saturation or significant changes in the general structure of the images.
Ablation study Fig. 20 displays visual example of unconditional generation with controlled "eeg and Consistent with results in Sec. 5.5, controlling image quality with has fewer side effects than controlling with Yseg.
D Pipeline figure
The overall pipeline and conceptual framework of SEG presented in Fig. 7. Fig. (a) and Fig. 7 (b) depict the original sampling process and the modified sampling process with smoothed energy, respectively, Fig. 7 (c) illustrates the the final prediction (the red arrow) with the guidance scale.
16

---

Vanilla SDXL
SAG
PAG
SEG (o = 10)
SEG (a - o0)
Figure 8: Qualitative comparison of SEG with vanilla SDXL [35], SAG [17], and PAG [1].
17

---

Vanilla SDXL
SAG
PAG
SEG (o = 10)
SEG (a - o0)
Figure 9: Qualitative comparison of SEG with vanilla SDXL [35], SAG [17], and PAG [1].
18

---

Condition
(SDXL)
o=1 1
0= 2
0=5 5
0 10
0 00
Figure 10: Conditional generation using ControlNet [51] and SEG.
Condition
0 (SDXL)
==1
== 2
0=5
10
8
Figure 11: Conditional generation using ControlNet [51] and SEG.
(SDXL)
a
10
Query Blurring
Key Blurring
Query Blurring
Key Blurring
Figure 12: Comparison between query and key blur across different values of a.
19

---

(SDXL) ==1 o=2
0=5
o 10
Figure 13: Unconditional generation using SEG.
20

---

(SDXL)
0=1
0=
0 ==5
10
"a Victorian-era pocket gears of candy, macro photogrophy"
"a Banksy-inspired groffiti child letting balloon shaped like Earth"
"an insect robot preparing deficious meal, anime style"
"A majestic oil painting of raccoon Queen wearing red French rayal gown. The painting is hanging on ornote wall decorated with wallpaper."
Figure 14: Text-conditional generation using SEG.
21

---

Yefg = 1
Yel
Yefg = 5
Yegg = 7
Yeig = 9
0= ==1 1
10
0
Figure 15: Experiment on the combination of SEG and CFG. Bee is fixed to 3.0. The prompt is "a friendly robot helping old lady cross the street." Without causing saturation or significant changes in the general structure, SEG improves the image quality.
Yefg
Ydg
Yefg 5
Y 7
Yefg 9
== 10
0000
Figure 16: Experiment on the combination of SEG and CFG. Ysey is fixed to 3.0. The prompt "a skateboarding turtle zooming through mini city made of Legos."
22

---

Ydf =
Yefg =
Ydfg =
Ydf = 7
Yefg =
0=1 1
10
0
Figure 17: Experiment on the combination of SEG and CFG. Yseg is fixed to 3.0. The prompt is "a group of puppies playing soccer with a ball of yarn."
Ycfg =
Yofg
Yefg =
Yefg = 7
Ydg 9
==1
0=
00
Figure 18: Experiment on the combination of SEG and CFG. Tsep is fixed to 3.0. The prompt is "a family of teddy bears having barbecue in their backyard."
23

---

Yefg =
Ydf
Yelg =
Ycgg =
Ydg =9 9
==1
10
0000
Figure 19: Experiment on the combination of SEG and CFG. Yheg is fixed to 3.0. The prompt is "a baby elephant learning to paint with its trunk in art studio."
0=1
0 ==2
O 5=5
10
Yseg = 1
Yseg = 2
Yseg = 5
Yseg = 10
Yseg 100
Figure 20: Unconditional generation result with controlled 7sng and a.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction state the main claims and contributions of the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
The paper provide the full set of assumptions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
The paper provide open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct http:///ggiieeeeeeeuuuubiiiiiiii
/CodeSubmic ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports statistical information.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerp//.///ee cc
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss potential societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper proposes guidance method for the current model; therefore, the paper itself poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi tecodeccoo.e.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
29

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "dcWXUpTYwi", "reviewer": "Reviewer_2oF8", "summary": "The method (SEG) discussed in the paper mainly applies an energy-based optimization on the emerging values in the self-attention to reduce the curvature of the energy landscape of attention, leading to improved image quality and less structural change from the original prediction compared to previous approaches. SEG is training- and condition-free and can be used for both unconditional and conditional sampling strategies. The authors validate the effectiveness of SEG by evaluating generated images with and without text conditions, as well as with ControlNet.", "questions": "What is the advantage of your method compared to other recent works in text-to-image generation that is mentioned in the weaknesses?", "limitations": "The validation can be improved.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper looks at the refinement of image generation through emerging values in attention layers, which is very interesting. The paper investigated multiple conditions for the diffusion model they used, and the visualizations are insightful.", "weaknesses": "The quantitative evaluations needed to be presented in a more detailed manner.\nThe number of works they compared against could be much more.\ne.g. imagic (Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I. and Irani, M., 2023. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6007-6017).), LEdit++ (Brack, M., Friedrich, F., Kornmeier, K., Tsaban, L., Schramowski, P., Kersting, K. and Passos, A., 2024. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8861-8870). ) or collaborative diffusion (Huang, Z., Chan, K.C., Jiang, Y. and Liu, Z., 2023. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6080-6090).),", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank the reviewer for acknowledging our visualization with various conditions as insightful and our paper as very interesting. We'd like to address the concerns and questions raised:\n\n## Additional details of quantitative evaluations\n\n> While we guide readers more towards qualitative assessment, we do employ multiple approaches to evaluate image quality. Quantitatively, we use FID and CLIP scores to measure fidelity of samples. To assess unintended side effects, we utilize LPIPS scores to quantify deviations from unguided images. Qualitatively, we present extensive visual comparisons that demonstrate improvements in definition, expression, sharpness of details, realism of textures, and overall composition. As Reviewer CTSN has mentioned, we believe this approach provides a comprehensive evaluation of our method's effectiveness and demonstrate our paper's point on the empirical side.\n\n> Besides, as mentioned in a concurrent work [A], the FD-DINOv2 metric is another means to calculate Fréchet distances and is well-aligned with human perception. For reference, in the table below, we present FD-DINOv2 scores calculated using 50k samples from the EDM2-S model trained on ImageNet-64 to assess fidelity. We also include uncurated qualitative samples from this model in Fig. 3 of the attached PDF. This corroborates how the structure and quality of samples change, as well as the generality of our methods.\n| Model | FD-DINOv2$\\downarrow$ |\n| --- | --- |\n| No guidance | 95.1915 |\n| SEG | **47.4733**\n\n[A] Karras, Tero, et al. \"Analyzing and improving the training dynamics of diffusion models.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n## Compare against more recent works (e.g. Imagic, LEdits++, collaborative diffusion)\n\n> We respectfully would like to emphasize that our method differs from recent text-based editing works like Imagic, LEdits++, and collaborative diffusion in its generality and goal. Our approach is more general, aiming to improve quality for diffusion models equipped with self-attention, regardless of the presence of text conditions.\n\n> However, we recognize that these text-based editing methods are orthogonal to our method and could be potential applications of our approach. In the revised manuscript, we may include a discussion of the potential for synergistic applications in the future direction part.\n\n## Address the question about advantages over recent works\n\n> We appreciate the opportunity to clarify this point. While our method is complementary to those text-based editing works, our method features several characteristics:\n\n> 1. Generality: Unlike text-based editing methods, our approach can improve image quality without requiring text prompts, making it applicable to a broader range of scenarios. SEG demonstrates effectiveness in both conditional and unconditional image generation scenarios, making it a flexible solution (Reviewers 2oF8 and cNNp).\n2. Inference-time quality enhancement: SEG improves the overall definition, expression, sharpness of details, realism of textures, and overall composition of generated images across various conditions, e.g., no condition, text, Canny, or depth map (Reviewers 2oF8 and Rhhf). Text-based editing methods the reviewer mentioned do not target and induce quality improvement.\n3. Novel approach: Our method, Smoothed Energy Guidance (SEG), presents an innovative, training- and condition-free approach to image generation using diffusion models. It offers an interesting alternative to classifier-free guidance (CFG) by leveraging the self-attention mechanism (Reviewers 2oF8, Rhhf, 9qC6, and CTSN).\n4. Theoretical foundation: The paper provides a solid theoretical grounding, using an energy-based perspective and the concept of smooth energy landscapes to improve image generation (Reviewers Rhhf and CTSN).\n\n## General response and additional figures\n> We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.\n\nWe will elaborate on these points in the revised manuscript to clearly articulate the unique benefits of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ceAZnxG2H", "reviewer": "Reviewer_Rhhf", "summary": "The manuscript introduces SEG, a novel training- and condition-free guidance method for enhancing image generation with diffusion models. The method leverages an energy-based perspective of the self-attention mechanism and introduces a technique to reduce the curvature of the energy landscape of attention, thereby improving the quality of generated images. SEG controls the guidance scale by adjusting a Gaussian kernel parameter, offering a flexible and theoretically grounded approach to unconditional and conditional image generation. The authors validate the effectiveness of SEG through extensive experiments on conditions, showcasing its superiority over existing methods like SAG and PAG in terms of sample quality and reduction of unintended effects. My detailed comments are as follows:", "questions": "(1) The author proposes that the model can generate images without giving any conditions, so how does the model know what to generate?\n(2) In Figure 2, as σ increases, the image becomes clearer, but the style is completely different from the initial one. Why is this? How to determine the optimal value of σ?\n(3) What are the main application scenarios of this model? How to generate the image that the user wants without giving any conditions?\n(4) In Figure 5, Qualitative comparison of SEG with vanilla SDXL, SAG, and PAG. However, the image is labeled as PEG. Please provide a detailed explanation. Also, the full name of PEG is not given.\n(5) In the manuscript, Smoothed Energy Guidance (SEG) is defined multiple times. Generally, abbreviations should be defined the first time they appear and used consistently in subsequent content. Please also check other abbreviations.", "limitations": "There is a lack of detailed discussion, such as how to determine the value of σ and why increasing the value of γ does not improve the sample quality in terms of FID and CLIP scores.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This article proposes a novel training- and condition-free image generation method, SEG, which significantly improves diffusion models through the theoretical foundation of smooth energy landscapes and the introduction of Gaussian blur on attention weights and the development of efficient query blur techniques. The quality of the images generated and their superiority verified on multiple conditions.", "weaknesses": "The method proposed in this manuscript, SEG, relies heavily on the quality of the baseline model and may amplify biases or harmful stereotypes in existing data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review of our paper and the recognition of SEG's strengths. We'd like to address the concerns and questions raised:\n\n## How the model knows what to generate and why the style is different\n> First, we'd like to note that our method claims an inference time boost like other guidance techniques, e.g., CFG, by modulating and utilizing the modeled distribution. On a high level, samples from the blunter distribution have poor definition, expression, sharpness of details, realism of textures, and overall composition. When we use these as the negative prediction of the score for guidance, we gain samples without such properties.\n\n> Concretely, we can explain how SEG guides the sampling process by drawing an analogy to CFG and from a probabilistic perspective:\n\n> a) Similar to how CFG uses the difference between predictions based on sharper conditional and smoother unconditional distributions, SEG reduces the curvature of the energy landscape underlying self-attention (Theorem 3.1). This creates a smoother landscape for attention minimization during sampling, enhancing sample quality without relying on external conditions or special training.\n\n> b) From a probabilistic perspective, the energy is associated with the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood (Lemma 3.2) and reduces the distribution's curvature (Theorem 3.1).\n\n> The style changes with increasing $\\sigma$ you mention seem like changes in the realism of textures or colors, and this is also an effect of using a blunter distribution as negative guidance. Still, our method does not change vanilla SDXL more than PAG or SAG in terms of qualitative and quantitative metrics while achieving a significant quality boost. For example, PAG significantly changes the color, style, and structure of the original SDXL output, as shown in Figs. 5, 7, and 8.\n\n## Discussion on parameters\n> In Fig. 1 of the attached PDF, we demonstrate experiments with controlled $\\sigma$ and $\\gamma_\\mathrm{seg}$, which support our quantitative experiment and claim in Sec. 5.5 and Fig. 6 in the main paper. Larger $\\gamma_\\mathrm{seg}$ values can cause side effects such as saturation by pushing pixels linearly, potentially moving them out of the manifold, as can be inferred from Fig. 4(c) of the attached PDF. In contrast, Gaussian blur with $\\sigma$ naturally smoothens the energy in diffusion models and yields the score prediction which is more likely to exist on the manifold, causing benign predictions even if $\\sigma$ gets infinitely big.\n\n> Also, we chose optimal $\\sigma$ values ($10$ and $\\infty$) based on perceptual quality (in Figs. 2, 3, and 4) and metrics (FID and CLIP Score in Table 2). We found that in most cases, to generate realistic photos with better structure and composition, $\\sigma \\to \\infty$ is the best choice, while for generating aesthetic images, $\\sigma=10$ works well.\n\n## Application scenarios\n> SEG's main applications include enhancing both conditional and unconditional image generation across various domains. It has many applications in large-scale diffusion models trained on text-image pairs for unconditional and general generation. ControlNet, which uses conditions other than text, is one example. SEG can also be used for inverse problems.\n\n## Reliance on baseline models\n> While SEG does rely on the baseline model, this is inherent to all guidance methods, which aim to enhance or control the generation process during inference. We have clarified this limitation in the paper. Still, it can be seen as a strength that our theoretical results are general to all models and that SEG is used in both unconditional image generation and conditional generation with various types of input, as acknowledged by Reviewers 2oF8 and cNNp.\n\n## Abbreviations\n> We apologize for the labeling error in Fig. 5. \"PEG\" should indeed be \"SEG\". We will correct this in the final revision and ensure consistent use of abbreviations throughout.\n\n## General response and additional figures\n> We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5aCvnRfqo6", "reviewer": "Reviewer_9qC6", "summary": "This paper presents a method for unconditioned image generation based on Diffusion Models, specifically using Stable Diffusion. The proposed method offers an alternative to classifier-free guidance (CFG), eliminating the need to train a classifier for adding conditions. Traditionally, the CFG denoising function incorporates both conditional and unconditional terms, while unconditional generation only involves an unconditional evaluation of the U-Net. Instead, this method performs unconditional diffusion by introducing an additional term to the traditional reverse process.\nThis new term is an energy-based component calculated as a proportion of the Gaussian blurring of the self-attention layers in the U-Net. The core idea is to use a blurred version of the unconditional self-attention layers to shift the mean distribution of the original unconditional prediction, thereby \"smoothing\" the prediction.\nThe paper employs the standard SDXL model and compares its results against the Perturbed Attention Guidance (PAG) and Self-Attention Guidance (SAG) methods.", "questions": "The questions are listed in the explanation of the weaknesses. It is encouraged to provide further explanation to the presented concerns to have the chance of increasing this initial score. \nIn general, the paper, while presenting theoretical justifications, needs clearer explanations of its claims, particularly concerning Gaussian curvature attenuation and the linear mapping property in blurring queries. The impact of the sigma parameter on image saturation requires more discussion, including balancing sigma and gamma. The lack of comparisons with other diffusion models limits the assessment of the method's generalization, and the absence of perceptual metrics for Vanilla SDXL restricts validation to a few examples. While the FID disclaimer is noted, traditional community metrics should be used.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper presents an interesting way to use the unconditional prediction of the network as another way to do guidance generation. \n\nThis paper presents a method that aims to improve traditional unconditional prediction in diffusion models by smoothing the distribution through the addition of a term to the conventional unconditional image generation process. The proposed idea is analogous to the original CFG methodology in that it combines two different distributions to enhance the generated image. In this case, the smoothing of the distribution is achieved by incorporating a blurred version of the self-attention layers.", "weaknesses": "-  Technical correctness of the paper\n\nThe claim in lines 191-192, where the paper asserts that $\\sigma \\rightarrow 0$ remains the same as the original [image], is not entirely clear, and the intuition behind this is not thoroughly presented. The concern arises because Eq. 7 indicates that the proposed method involves adding another term to the original unconditional generation. According to the literature [10], what this paper refers to as $s_\\theta(x,t)$ corresponds to a distribution, as does $\\tilde{s_{\\theta}}(x,t)$. However, having $\\sigma$ approaching to 0 does not imply that this term becomes 0; rather, it means that the Gaussian filter affecting the calculation of the self-attention is 0. Therefore, the inference drawn from this reasoning is that if the filter's variance approaches 0, then $s_\\theta(x,t)$ approximates to $\\tilde{s}_\\theta(x,t)$.\n\nSince this paper relies on a strong theoretical background, it is crucial to provide a thorough demonstration and explanation of all claims. For instance, in line 150, what does it mean to be under a \"reasonable\" assumption to demonstrate the attenuation of the Gaussian curvature? Additionally, what does the paper refer to when discussing the property of linear mapping in the blurring of the queries between lines 167 and 169?\n\nAs the method uses a Gaussian kernel for the blurring process, it involves a sigma parameter derived from the Gaussian filter. However, there is insufficient discussion on how this parameter significantly affects the saturation of the generated image. From line 211, it can be interpreted that sigma controls the saturation, but between lines 274-276, the paper indicates that saturation can be altered by using the guidance scale for this method. The question is how sigma and gamma can be balanced to avoid over-saturated images and whether other characteristics of the image (brightness, darkness, vibrance, etc.) can be controlled through these parameters.\n\n- Experimental validation\n\nThis paper limits its exploration by not providing further comparisons using other diffusion models as backbones. This omission raises questions about the proposed method's generalization capability.\n\nIt is not clear why the paper does not report the perceptual metric for the Vanilla SDXL in Table 1.\nThe lack of a metric to compare the perceptual performance of the Vanilla SDXL against the proposed method raises a concern, as the validation is then limited to just a few image examples presented in this manuscript. Without this measure, it is difficult to fully understand the overall performance of the proposed method compared to the baseline and other state-of-the-art methods.\n\nThe disclaimer in lines 231–232 regarding the FID is acknowledged. However, when it comes to image generation, these are the metrics that the community has adopted and learned to interpret. Of course, this does not limit new research from proposing new ways to measure image generation to provide a \"strict measure of image quality\" and other attributes.\n\nThis paper claims (line 14-15) that its implementation does not significantly increase computational cost. However, it is unclear how the computational requirements should be adjusted to use it. Moreover, the paper directly compares its method against unconditional generation, which only requires a single network evaluation, while this method (according to Eq. 7) requires at least two network evaluations. Furthermore, using a conditional input would necessitate at least three network evaluations, likely leading to a considerable increase in computational cost. It is suggested that the paper include an analysis of the trade-off in terms of the number of function evaluations (NFEs) to fully understand how many network evaluations are needed and how this method impacts computational cost.\n\nIt would have been interesting to see how other filter kernels perform besides Gaussian blurring.\n\n- Presentation \n\nAlthough this paper is well-written and presented, its intelligibility could be improved by adding a few more sentences to clarify some of the intuitions described above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review of our paper. Below is our response to your review:\n\n## When sigma approaches 0\n> SEG with $\\sigma\\to 0$ is equivalent to the original sampling process. This doesn't necessarily mean $\\\\tilde{s}\\_\\\\theta(x, t)$ goes to 0, but rather that the Gaussian kernel becomes a Dirac delta kernel, implying a single value of 1 in the center after discretization and normalization. Filtering with this kernel is an identity operation, so $s\\_\\\\theta(x, t) = \\\\tilde{s}\\_\\\\theta(x, t)$ as the filtering has no effect. Replacing $\\\\tilde{s}\\_\\\\theta(x, t)$ with $s\\_\\\\theta(x, t)$ in Eq. 7 yields:\n$$dx=[f(x,t)-g(t)^2(\\gamma_\\mathrm{seg}s_\\theta (x, t)-(\\gamma_\\mathrm{seg}-1)s_\\theta(x, t))]dt+g(t)d\\bar{\\omega}=[f(x, t)-g(t)^2(s_\\theta (x, t))]dt+g(t)d\\bar{\\omega},$$\nsince the gamma $\\gamma_\\mathrm{seg} {s}_\\theta (x, t)$ terms cancel out each other.\n\n## Explanation on claims\n> We thank the reviewer for raising this point. Although we already have the basic explanation on claims in the Appendix, we would like to clarify more.\n\n> Let $\\mathbf{a}=(a_1,\\ldots,a_n)$ denote the attention values before the softmax operation, and let $\\tilde{\\mathbf{a}} = (\\tilde{a}_1,\\ldots,\\tilde{a}_n)$ denote the attention values after applying the 2D Gaussian blur. Let $H$ denote the Hessian of the original energy, \\textit{i.e.}, the derivative of the negative softmax, and $\\tilde{{H}}$ denote the Hessian of the underlying energy associated with the blurred weights.\n\n> The elements in the $i$-th row and $j$-th column of the Hessian matrices are given by:\n$$\nh\\_{ij}=(\\xi(\\mathbf{a})_i-\\delta\\_{ij})\\xi(\\mathbf{a})_j,\\quad\\tilde {h}\\_{ij}=(\\xi(\\tilde{\\mathbf{a}})_i-\\delta\\_{ij})\\xi(\\tilde{\\mathbf{a}})_j b\\_{ij},\n$$\nrespectively, where $b\\_{ij}$ are the elements of the Toeplitz matrix corresponding to the Gaussian blur, and $\\delta\\_{ij}$ denotes the Kronecker delta.\n\n> Assuming $\\\\xi(\\\\tilde{a})\\_i \\\\xi(\\\\tilde{a})\\_j \\\\approx 0$ and $\\\\xi(a)\\_i \\\\xi(a)\\_j \\\\approx 0$ for all $i$ and $j$, which is a reasonable assumption when the number of tokens is large and the softmax values get small, the non-diagonal elements of the Hessians approximate to 0 and the diagonal elements dominate. The determinants of the Hessian matrices are approximated as:\n$$|\\\\det(H)|\\\\approx\\\\prod\\_{i=1}^n\\\\xi(a)\\_i,\\\\quad   |\\\\det(\\\\tilde{H})| \\\\approx \\\\prod\\_{i=1}^n \\\\xi(\\\\tilde{a})\\_i b\\_{ii}$$\n\n> We have the following inequality:\n$$\\\\prod\\_{i=1}^n\\\\xi(\\\\tilde{a})\\_i b\\_{ii}<\\\\prod\\_{i=1}^n\\\\xi(\\\\tilde{a})\\_i=(e^{\\\\sum\\_{j=1}^n \\\\tilde{a}\\_j}) / (\\\\sum\\_{j=1}^n e^{\\\\tilde{a}\\_j})^n\\\\leq (e^{\\\\sum\\_{j=1}^n a\\_j})/(\\\\sum\\_{j=1}^ne^{a\\_j})^n=\\\\prod\\_{i=1}^n\\\\xi(a)\\_i,$$\nwhere the first inequality follows from the property of the Gaussian blur kernel, $0 \\\\leq b\\_{ii} < 1$, and the second inequality is derived from Lemmas 3.1 and 3.2, which demonstrate the mean-preserving property and the decrease in the lse value when applying a blur. The monotonicity of the logarithm function implies that the denominator involving the blurred attention weights is smaller. Eventually, we obtain the following inequality:\n$$|\\\\det(\\\\tilde{H})|<|\\\\det(H)|$$\nThis implies that the updated value is derived with attenuated Gaussian curvature ($K=\\\\det(\\\\tilde{H})$ in our case), of the energy function underlying the blurred softmax operation compared to that of the original softmax operation.\n\n## Why it does not present perceptual metric for Vanilla SDXL\n> This is because the LPIPS metric is calculated with Vanilla SDXL (line 229), to measure the side effects (how much the sampling process with guidance deviates from those of the original sampling process). You can recognize the perceptual distance as zero, since it does not differ from the original sampling process.\n\n## Discussion on parameters\n> In Fig. 1 of the attached PDF, we present samples with controlled $\\sigma$ and $\\gamma_\\text{seg}$, which support our experiment in Sec. 5.5 and Fig. 6 and our claim in the main paper. Contributing to saturation, large $\\gamma_\\text{seg}$ values linearly push the pixel values far from the original prediction.\n\n## Additional details of quantitative evaluations\n> While we guide readers more towards qualitative assessment, we employ multiple approaches to evaluate image quality. Quantitatively, we use FID and CLIP scores to measure fidelity of samples. To assess unintended side effects, we utilize LPIPS scores to quantify deviations from unguided images. Qualitatively, we present extensive visual comparisons (e.g., Figures 2-5, 7-10) that demonstrate improvements in definition, expression, sharpness of details, realism of textures, and overall composition. This approach demonstrates our paper's main contribution, as Reviewer CTSN has mentioned.\n\n> Besides, as mentioned in a concurrent work [A], the FD-DINOv2 metric is another means to calculate Fréchet distances and is well-aligned with human perception. For reference, in the table below, we present FD-DINOv2 calculated using 50k samples from the EDM2-S model trained on ImageNet-64 to assess fidelity. We also include uncurated qualitative samples from this model in Fig. 3 of the attached PDF.\n| Model | FD-DINOv2$\\downarrow$ |\n|-|-|\n| No guidance | 95.1915 |\n| SEG ($\\sigma \\to \\infty$) | **47.4733** |\n\n[A] Karras et al. \"Analyzing and improving the training dynamics of diffusion models.\" CVPR 2024.\n\n## Computational cost\n> Our NFE is the same as other methods, SAG and PAG, which is 50 per sample. The contribution part in complexity is the blurring operation, which incurs quadratic cost in the number of tokens. The query blurring instead blurs the query matrix rather than the attention map (shown as equivalent in lines 170-176), so it avoids the quadratic complexity and it's our contribution that makes the blurring process feasible in the high resolution situation.\n\n## General response and additional figures\n> We respectfully refer the reviewer to our general response and additional figures provided above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTZvCN9vXN", "reviewer": "Reviewer_CTSN", "summary": "The paper proposes a technique to improve unconditional sampling from diffusion models. The main idea is to translate the notion of classifier-free guidance (CFG) to the case in which there is no condition available. To this end, the paper notes that the conditional prediction is \"sharp\", while the unconditional prediction is \"smooth\", or more simply put, the unconditional prediction is smoother than the conditional one. The paper also notes that applying Gaussian filtering on the attention weights of the model's unconditional prediction yields an (even) smoother prediction. The paper then combines these two observations, and proposes SEG, a version of CFG that is applicable to the unconditional case: CFG requires the conditional and unconditional predictions; SEG replaces the conditional prediction with the unconditional, while replacing the unconditional with its smoothed version. By doing this procedure, SEG manages to preserve the sharpness-smoothness relation that CFG has between the factors (the conditional and unconditional predictions).\n\nThe paper then demonstrates that this procedure translates into improved performance compared to reasonable competitors on standard benchmarks.", "questions": "- I can understand that the main interest of the paper was improving unconditional generation. The paper achieves this objective by, in a sense, computing a less sharp/more smooth prediction than the unconditional one; this prediction is combined with the original unconditional prediction in the same way that CFG operates. I think that approach is interesting from the point of view of efficiency: a useful prediction is being obtained by an inexpensive operation (i.e. smoothing), instead of another forward pass. Given this context, doesn't it make sense to rather aim at improving conditional generation? (which is arguably the most important one). I think the paper's proposal could be used directly to improve the efficiency conditional generation with CFG: do the conditional forward passes, and replace the unconditional ones with the ones that result from Gaussian smoothing (that is, I think you can replace an entire function evaluation with simple Gaussian smoothing of the conditional prediction). Have the authors considered this option? \n[I think Eq. (8) with gamma_seg=0 is somewhat similar, but I'd be thinking of replacing s(x,t) with a \\tilde{s}(x,t) = smooth(s(x,t,c))]", "limitations": "Yes, they have", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a simple yet effective procedure to improve unconditional sampling in diffusion models. The idea, in my view, is elegant\n- The paper is, over all, well written and clear\n- Both qualitative and quantitative evaluations seem to demonstrate the paper's point on the empirical side\n- I think the paper's proposal could be used for improving the efficiency of conditional sampling, which is arguably an even more useful case", "weaknesses": "## Weaknesses that don't affect my rating\n- The Method section could strongly benefit from a figure illustrating on what object, exactly, is the Gaussian blur being applied. The paper currently doesn't have a \"pipeline\" figure\n- I understand the value of Lemma 3.1, but I don't think it's necessary to provide a proof in the main paper. As a side note, I don't even think it's necessary to provide a proof: as far as I understand, this fact is widely known, since the Gaussian filter is normalized and symmetric, no?\n  - For instance, I think a well-known intuition in the computer vision community is that (infinite) successive applications of Gaussian blurring on an image result in an image with a single color, that corresponds to the average color of the initial image (i.e. same mean, and zero variance).\n- L188 mentions PAG. It would be useful to remind the reader what it stands for. (I think it was originally just mentioned in the introduction)\n- Unless I'm misunderstanding, L211 claims that, contrary to two other methods--SAG and PAG--the presented method is both training- and condition-free. I'm not sure I follow: aren't those two methods both also training- and condition-free?\n- I understand what the authors mean in L231 by \"FID has been found not to be a strict measure of image quality\". However, FID in itself does not measure image quality, but distance w.r.t. some distribution\n------------\n## Weaknesses that don't affect my rating, but should be addressed:\n- Eq. (5): please consider explaining that \"lse\" standards for logsumexp a bit before L85: by then it's a bit confusing already\n- Isn't L86 redundant (since the same statement is made--and referenced--in L75?)\n- L95: please consider denoting the LSE with some convention: just writing \"lse\" lends itself to confusion\n- L97: a negative prediction of what?\n- L108: to make which operation tractable?\n- L124: increases the LSE or decreases (according to L127). My intuition is that it decreases, since LSE approximates the maximum of the tensor\n- The proof of proposition 3.1 could be made shorter, or sent to the appendix\n- L177: \"Note that the Gaussian blur can also be applied to K\" I presume this is true because of the commutation property of the convolution operation. Perhaps explicitly stating that in the paper would be better\n\nTypos and such:\n- L9: actually?\n- L39: why \"actually\"?\n- L70: conditioning on\n- L72: demonstrates its prevalence? sounds weird\n- L111 and equation at L 113: boldface \"a\"?\n- Minor comment for all over the paper: use \\eqref for equations\n- L230: \"how much guided images are altered from unguided ones\" sounds a bit weird\n- L232: \"being more favorable to users\" sounds weird\n- Fig. 2 and 11: I think there's a repeated sample? (the one indoors)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your acknowledgment of our approach's strengths, particularly its elegant idea and thorough qualitative and quantitative evaluations. Thank you for your careful suggestions. We'd like to address the concerns and questions you've raised:\n\n## Improving conditional generation\n> We'd like to highlight that our method improves conditional generation even without CFG, thanks to the smoothed energy curvature. When used in its conditional version as a replacement for CFG, rather than being used in combination with CFG, it reduces the number of function evaluations by one. We conducted experiments with the class-conditional EDM2-S model trained on ImageNet-64. For reference, the table below presents FD-DINOv2 scores calculated from 50k samples using this model to assess fidelity. We've also included uncurated qualitative samples in Fig. 3 of the attached PDF.\n| Model | FD-DINOv2$\\downarrow$ |\n|-------|-----------|\n| No guidance | 95.1915 |\n| SEG ($\\sigma \\to \\infty$) | **47.4733** |\n\n> It's important to note that our method doesn't propose to blur the output directly, and Gaussian blur on score prediction itself reduces the noise level. Instead, SEG applies Gaussian blur to attention weights. This process still requires a partial forward pass, even when reusing features before the attention weights for efficiency, which incurs slightly more overhead than using the same prediction twice.\n\n> In addition, when it comes to general text-to-image generation like Stable Diffusion, the interface between the given caption and generated image is mostly the cross-attention. However, even though our method can be applied to cross-attention, and even temporal attention (in text-to-video generation, which indeed works to improve consistency), we deal only with self-attention in this paper. We believe the ideas you suggested are promising future directions, though, and those directions will be emphasized in our revised version.\n\n## Pipeline figure\n> We've included a draft of the pipeline figure (Fig. 4 in the attached PDF) and will incorporate this in the revision. This includes where the Gaussian blur with $\\sigma$ is applied, the original and blurred attention weights, the associated energy landscape, and how the score predictions are obtained. We also include in Fig. 4(c) how the linear extrapolation between those predictions in SEG works.\n\n## Adding intuition to Lemma 3.1\n> While those familiar with Gaussian blur in the computer vision community may intuitively know this fact, we thought a clearer explanation was needed. However, we also find it intuitive that Gaussian blur preserves the mean while decreasing variance, since applying Gaussian blur to images causes the pixels to converge towards a similar value, reducing variance. Therefore, we will consider moving the formal proof to the appendix and instead include this intuition in the main text.\n\n## Definition of FID\n> We understand your point about FID. While it doesn't directly measure image quality, we use the metric as a proxy for realism and quality because it measures the distance to the real image-text distribution of the COCO dataset. We'll clarify this in the final revision.\n\n## Other concerns and suggestions\n> We acknowledge the typo in line 211 and confirm that SAG and PAG are indeed training- and condition-free. Also, we've incorporated most of your suggestions to improve the paper's clarity in our revised manuscript.\n\n## General response and additional figures\n> We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.\n\nThank you again for your valuable feedback. We believe these changes will help us fine-tune the quality and clarity of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBAkcbQBlE", "reviewer": "Reviewer_cNNp", "summary": "The paper presents a condition-free guidance method for diffusion models. The guidance is generated from the self-attention mechanism to perform guidance from an energy-based perspective as an alternative to classifier-free guidance. With this, the work aims to train the models for improved quality performance in conditional and unconditional image generation.", "questions": "Questions were mentioned in the weaknesses section. Additionally, it would be recommended to introduce the relationship between reducing the curvature of the enemy function and guidance for context.", "limitations": "The paper discusses potential limitations and societal impact of generative models in the conclusion section.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The model presents an alternative guidance method for the diffusion model that is independent of an explicit condition, allowing it to work with conditional and unconditional generation.", "weaknesses": "As a suggestion by the paper is to refer to a qualitative assessment, what would be the criteria to consider a model or ablation generates a better quality?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank the reviewer for acknowledging the strength of our approach in its versatility. We demonstrate SEG's effectiveness in both unconditional and conditional settings, including text-conditional generation and ControlNet conditioning. This flexibility allows SEG to improve image quality across various generation tasks without requiring task-specific modifications. Additionally, we appreciate the reviewer's thoughtful comments and questions. We would like to address the main points raised.\n\n## Criteria for assessing image quality\n> While we guide readers more towards qualitative assessment, we do employ multiple approaches to evaluate image quality. Quantitatively, we use FID and CLIP scores to measure sample quality. To assess unintended side effects, we utilize LPIPS scores to quantify deviations from unguided images. Qualitatively, we present extensive visual comparisons (e.g., Figs. 2-5 and 7-10 in the main paper) that demonstrate improvements in definition, expression, sharpness of details, realism of textures, and overall composition. We believe this multi-faceted approach provides a comprehensive evaluation of our method's effectiveness. For overall improvement, we have also included uncurated samples from the Vanilla SDXL model, both without and with SEG, in Fig. 2 of the attached PDF.\n\n> Besides, as mentioned in a concurrent work [A], the FD-DINOv2 metric is another means to calculate Fréchet distances and is well-aligned with human perception. For reference, in the table below, we present FD-DINOv2 scores calculated using 50k samples from the EDM2-S model trained on ImageNet-64 to assess fidelity. We also include uncurated qualitative samples from this model in Fig. 3 of the attached PDF. This corroborates how the structure and quality of samples change, as well as the generality of our methods.\nModel | FD-DINOv2$\\downarrow$\n--- | ---\nNo guidance | 95.1915\nSEG ($\\sigma\\to\\infty$) | **47.4733**\n\n[A] Karras, Tero, et al. \"Analyzing and improving the training dynamics of diffusion models.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n## Relationship between energy curvature reduction and guidance\n> We appreciate the suggestion to clarify this relationship. On a high level, CFG uses the difference between the prediction based on the sharper conditional distribution and the prediction based on the smoother unconditional distribution to guide the sampling process. By analogy, SEG reduces the curvature of the energy landscape underlying self-attention (Theorem 3.1). This creates a smoother landscape for a minimization step of attention during sampling, analogous to how classifier-free guidance uses the difference between conditional and unconditional distributions. By using the \"blunter\" prediction from this smoother landscape as negative guidance, SEG enhances sample quality without relying on external conditions or special training.\n\n> From a probabilistic perspective, this process can be thought of as maximizing the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood, as shown in Lemma 3.2, and also reduces the curvature of the distribution, as shown in Theorem 3.1.\n\n## General response and additional figures\n> We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.\n\nWe hope these clarifications address the reviewer's concerns and highlight the strengths and contributions of our work. We're happy to provide any additional information or clarifications if needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General response to reviewers

> We sincerely thank all the reviewers for their thorough evaluation and insightful feedback on our submission. We appreciate the recognition of our work's strengths and the constructive suggestions to fine-tune our manuscript. Based on the reviews, we'd like to highlight the following strengths of our paper:

> 1. **Novelty**: Our method, Smoothed Energy Guidance (SEG), presents an elegant, training- and condition-free approach to image generation using diffusion models. It offers an interesting alternative to other guidance methods, such as CFG, SAG, and PAG, by leveraging smoothed energy landscape of the self-attention mechanism (**Reviewers 2oF8, Rhhf, 9qC6, and CTSN**).
2. **Theoretical grounding**: The paper provides theoretical grounding, using an energy-based perspective and the concept of smooth energy landscapes to improve image generation (**Reviewers Rhhf and CTSN**).
3. **Versatility**: SEG demonstrates effectiveness in both conditional and unconditional image generation scenarios, making it a flexible solution (**Reviewers 2oF8 and cNNp**).
4. **Empirical validation**: Our work provides both qualitative and quantitative evaluations that demonstrate the method's effectiveness (**Reviewer CTSN**). The reviewers also noted the quality of our generated images, verified across various conditions (**Reviewers 2oF8 and Rhhf**).
5. **Clear presentation**: The paper is generally well-written and clear, with insightful visualizations that help convey our method's effectiveness (**Reviewers 9qC6 and CTSN**).

## Additional figures

> In the attached PDF file, we present qualitative results with controlled $\sigma$ and $\gamma_\mathrm{seg}$ (Fig. 1), which support our quantitative experiment and claim in Sec. 5.5 and Fig. 6 in the main paper. We also present more uncurated results (Fig. 2 and Fig. 3), results from the class-conditional model of EDM2-S (Fig. 3), and a conceptual figure of the overall pipeline (Fig. 4).

Meta-review: This paper received split recommendations after the post-rebuttal discussion. The rebuttal adequately addressed most reviewer concerns about technical correctness and experimental validation. The author-reviewer discussion was engaged, and most remaining doubts were clarified. Overall, the paper presents an interesting technique to improve unconditional sampling from diffusion models that can have a practical impact. Reviewer comments and corrections, particularly those expressed by Reviewer 2oF8, should be carefully included in the final version.
Decision: Accept (poster)