Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention

Authors: susung  hong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 18:08:19

Abstract: Conditional diffusion models have shown remarkable success in visual content generation, producing high-quality samples across various domains, largely due to classifier-free guidance (CFG). Recent attempts to extend guidance to unconditional models have relied on heuristic techniques, resulting in suboptimal generation quality and unintended effects. In this work, we propose Smoothed Energy Guidance (SEG), a novel training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to enhance image generation. By defining the energy of self-attention, we introduce a method to reduce the curvature of the energy landscape of attention and use the output as the unconditional prediction. Practically, we control the curvature of the energy landscape by adjusting the Gaussian kernel parameter while keeping the guidance scale parameter fixed. Additionally, we present a query blurring method that is equivalent to blurring the entire attention weights without incurring quadratic complexity in the number of tokens. In our experiments, SEG achieves a Pareto improvement in both quality and the reduction of side effects. The code is available at https://github.com/SusungHong/SEG-SDXL.

Main paper: 
 introduction: 
 Diffusion models [12, 45, 46] have emerged as a promising tool for visual content generation, producing high-quality and diverse samples across various domains, including image [38, 40, 42, 8, 13, 30, 2, 24, 9, 29, 34, 33, 4, 41, 5, 20, 22] , video [11, 50, 23, 18, 15, 3, 19, 44] , and 3D generation [36, 27, 6, 26, 49, 43, 48, 16] . The success of these models can be largely attributed to the use of classifier-free guidance (CFG) [14] , which enables sampling from a sharper distribution, resulting in improved sample quality. However, CFG is not applicable to unconditional image generation, where no specific conditions are provided, creating a disparity between the capabilities of text-conditioned sampling and sampling without text. This disparity results in a restriction in application, e.g., synthesizing images with ControlNet [51] without a text prompt (see the last two columns of Fig. 1 ).Recent literature [17, 1] has attempted to decouple CFG and image quality by extending guidance to general diffusion models, leveraging their inherent representations [25, 32, 17] . Self-attention guidance (SAG) [17] proposes leveraging the intermediate self-attention map of diffusion models to blur the input pixels and provide guidance, while perturbed attention guidance (PAG) [1] perturbs the attention map itself by replacing it with an identity attention map. Despite these efforts, these methods rely on heuristics to make perturbed predictions, resulting in unintended effects such as smoothed-out details, saturation, color shifts, and significant changes in the image structure when given a large guidance scale. Notably, the mathematical underpinnings of these unconditional guidance approaches are not well elucidated. In this work, we approach the objective from an energy-based perspective of the self-attention mechanism, which has been previously explored based on its close connection to the Hopfield energy [39, 31, 7] . Specifically, we start from the definition of the energy of self-attention, where performing a self-attention operation is equivalent to taking a gradient step. In light of this, we propose a tuning-and condition-free method that reduces the curvature of the underlying energy function by directly blurring the attention weights, and then leverages the output as the negative prediction. We call this method Smoothed Energy Guidance (SEG).SEG does not merely rely on the guidance scale parameter that cause side effects when its value becomes large. Instead, we can continuously control the original and maximally attenuated curvature of the energy landscape behind the self-attention by simply adjusting the parameter of the Gaussian kernel, with the guidance scale parameter fixed. Additionally, we introduce a novel query blurring technique, which is equivalent to blurring the entire attention weights without incurring quadratic cost in the number of tokens.We validate the effectiveness of SEG throughout the various experiments without and with text conditions, and ControlNet [51] trained on canny and depth maps. Based on the attention modulation, SEG results in less structural change from the original prediction compared to previous approaches [17, 1] , while achieving better sample quality. 
diffusion models: 
 Diffusion models [12, 45, 46] are a class of generative models that generate data through an iterative denoising process. The process of adding noise to an image x over time t ∈ [0, T ] is governed by the forward stochastic differential equation (SDE):EQUATIONwhere f and g are predefined functions that determine the manner in which the noise is added, and dw denotes a standard Wiener process.Correspondingly, the denoising process can be described by the reverse SDE:EQUATIONwhere ∇ x log p t (x) represents the score of the noisy data distribution and d w denotes the standard Wiener process for the reversed time.Diffusion models are trained to approximate the score function with s θ (x, t) ≈ ∇ x log p t (x). To generate an image based on a condition c, e.g., a class label or text, one simply needs to train diffusion models to approximate the conditional score function with s θ (x, t, c) ≈ ∇ x log p t (x|c) and replace ∇ x log p t (x) with it in the denoising process. To enhance the quality and faithfulness of the generated samples, classifier-free guidance (CFG) [14] is widely adopted. Accordingly, the reverse process becomes:EQUATIONHere, s θ (x, t) is learned by dropping the label by a certain proportion, and γ cfg is a hyperparameter that controls the strength of the guidance. Intuitively, CFG helps us to sample from sharper distribution by conditioning on a class label or text.The attention mechanism [47] , which has been widely adopted in diffusion models [12] , has been interpreted through the lens of energy-based models (EBMs) [31, 39, 7] , especially through its close connection with the Hopfield energy [7, 39] . In the modern (continuous) Hopfield network, the attention operation can be derived based on the concave-convex procedure (CCCP) from the following energy function [39] :EQUATIONwhere ξ ∈ R 1×d , X ∈ R N ×d , and lse stands for the log-sum-exp function, defined as lse(v) := log N i=1 e vi . The quadratic term acts as a regularizer to prevent ξ from exploding [39] , while -lse(Xξ ⊤ ) penalizes misalignment between X and ξ.Mathematically, it turns out that the attention mechanism is equivalent to the update rule of the modern Hopfield network [7, 39] . Specifically, inspired by the Hopfield energy in (4), and noticing that the first term depends on the attention weights, we propose the following energy function for entire self-attention weights in diffusion models: Definition 2.1 (Energy Function for Self-Attention). Let Q ∈ R (HW )×d be a matrix of query vectors and K ∈ R (HW )×d be a matrix of key vectors, where H, W , and d represent the height, width, and dimension, respectively. Let A ∈ R (HW )×(HW ) := QK ⊤ . The energy function with respect to entire self-attention weights in diffusion models is defined as:EQUATIONNote that to explicitly denote the spatial dimension, we use the subscript (x, y) to represent the index of a row or column of the matrices. Despite using the definition in (5) for the rest of the paper for simplicity, we additionally discuss the dual case, where we use the swapped indexing, in Appendix B.This view leads us to an important intuition: the attention operation can be seen as a minimization step on the energy landscape, considering that the first derivative represents the softmax operation which also appears in the attention operation. Building upon this intuition, we argue that Gaussian blurring on the attention weights modulates the underlying landscape to have less curvature, and we demonstrate this in the following sections by analyzing the second derivatives.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations are discussed the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our aim is to theoretically derive the effect of Gaussian blur applied on the attention weights, which in the end attenuates the curvature of the underlying energy function. Then, utilizing this fact, we develop attention-based drop-in diffusion guidance that enhances the quality of the generated samples, regardless of whether an explicit condition is given. In Section 3.1, we claim some useful properties of Gaussian blur: that it preserves mean, reduces variance, and thus decreases the lse value. In Section 3.2, we find that the curvature of the energy landscape is attenuated by the attention blur operation, leading naturally to a blunter prediction for guidance. And finally, in Section 3.3, built upon this fact, we define Smoothed Energy Guidance (SEG) and propose the equivalent query blurring method, which can perform attention blurring while avoiding quadratic complexity in the number of tokens.In this section, we derive some important properties of the Gaussian blur with the aim of figuring out the variation of the energy landscape. To this end, we start from some mathematical underpinnings on applying Gaussian blur to attention weights.A 2D Gaussian filter is a convolution kernel that uses a 2D Gaussian function to assign weights to neighboring pixels. The 2D Gaussian function is defined as:G(x, y) = 1 2πσ 2 e -(x-µx ) 2 +(y-µy ) 2 2σ 2where µ x and µ y are the means in the x and y directions, and σ is the standard deviation. The 2D Gaussian filter possesses symmetry, i.e., G(x, y) = G(-x, -y), and normalization, i.e., G(x, y)dxdy = 1. In practice, we use a discretized version of the Gaussian filter with a finite kernel size depending on σ, normalized to sum to 1. Lemma 3.1. Spatially applying a 2D Gaussian blur to the attention weights a := Qk ⊤ preserves the average E i,j [a (i,j) ]. In addition, the variance monotonically decreases every time we apply the Gaussian blur.Proof sketch. Applying a 2D Gaussian filter to the attention weights a (i,j) yields the blurred values ã(i,j) :ã(i,j) = k m=-k k n=-k G(m, n) • a (i+m,j+n)where k is the filter size, G(m, n) is the Gaussian filter value at position (m, n), and a (i+m,j+n) is the attention weight at position (i + m, j + n). Since the Gaussian filter is symmetric and normalized, it can be shown that the mean of the blurred attention weights is equal to the mean of the original attention weights. Similarly, we can show that the variance monotonically decreases when we apply a 2D Gaussian filter. See Appendix A.1 for the complete proof.Note that this fact also implies that blurring with a Gaussian filter with a larger standard deviation causes a greater decrease in the variance of attention weights. This is because a Gaussian filter with a larger standard deviation can always be represented as a convolution of two filters with smaller standard deviations, due to the associativity of the convolution operation.Finally, we show that applying a 2D Gaussian blur to attention weights increases the lse value in (5), i.e., increases the energy in (5) . This provides a bit of intuition about the underlying energy landscape, yet it is more prominently utilized in the claims in the following sections. Lemma 3.2. Applying a 2D Gaussian blur to attention weights a := Qk ⊤ increases the lse term when we consider the second-order Taylor series approximation of the exponential function around the mean µ := E i,j [a (i,j) ]. Consequently, the maximum is achieved when the attention is uniform, i.e., a (i,j) = a (k,l) ∀i, j, k, l. This corresponds to the case when we apply the Gaussian blur with σ → ∞.Proof sketch. Applying the second-order Taylor series approximation around the mean µ, and using Proposition 3.1, we show that the second-order approximation of lse(a) is larger than or equal to that of lse(ã). Subsequently, we introduce Lagrange multipliers to find the maximum, which gives us the result, a (i,j) = a (k,l) ∀i, j, k, l. We leave the full proof in Appendix A.2.In this section, we demonstrate that applying a 2D Gaussian blur to the attention weights before the softmax operation results in computing the updated value with reduced curvature of the underlying energy function. To this end, we analyze the Gaussian curvature before and after blurring the attention weights. This is closely related to the Hessian of the energy function.Theorem 3.1. Let the attention weights be defined as a := Qk ⊤ . Consider the energy function in (5) . Then, applying a Gaussian blur to the attention weights a before the softmax operation results in the attenuation of the Gaussian curvature of the underlying energy function where gradient descent is performed.Proof sketch. Let H denote the Hessian of the original energy function, i.e., the derivative of the negative softmax, and H denote the Hessian of the new energy function associated with blurred attention weights. Furthermore, let b ij denote the i-th row, j-th column entry in the Toeplitz matrix B representing the Gaussian blur. Calculating the derivatives, we have the elements of the Hessians, To provide more intuition about what is actually happening and how we utilize this property in the later section, it is intriguing to consider the attenuating effect on the curvature in analogy to classifier-free guidance (CFG). CFG uses the difference between the prediction based on the sharper conditional distribution and the prediction based on the smoother unconditional distribution to guide the sampling process. By analogy, we propose a method to make the landscape of the energy function smoother to guide the sampling process, as opposed to the original (sharper) energy landscape.h ij = (ξ(a) i -δ ij )ξ(a) j and hij = (ξ(ã) i -δ ij )ξ(ã) j b ij . Using Lemmas 3.From a probabilistic perspective, the energy is associated with the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood as shown in Lemma 3.2, and also reduces the curvature of the distribution as shown in Theorem 3.1.Based on the above observation that the Gaussian blur on attention weights attenuates the curvature of the energy function, we propose Smoothed Energy Guidance (SEG) in this section. For brevity, we redefine the unconditional score prediction as s θ (x, t), and the unconditional score prediction with the energy curvature reduced as sθ (x, t). Specifically, sθ (x, t) is the prediction with the attention weights blurred using a 2D Gaussian filter G with the standard deviation σ. We formulate the process as:EQUATIONwhere * denotes the 2D convolution operator. Then, we replace the original attention weights with (QK ⊤ ) seg and compute the final value as in ordinary self-attention.For practical purposes when the number of tokens is large, we propose an efficient computation of (6) using the property of a linear map, since the convolution operation is linear. Concretely, blurring queries is exactly the same as blurring the entire attention weights, and we propose the following proposition to justify our claim. Proposition 3.1. Let Q and K be the query and key matrices in self-attention, and let G be a 2D Gaussian filter. Blurring the attention weights with G is equivalent to blurring the query matrix Q with G and then computing the attention weights.Proof. Since the convolution operation is linear, we can always find a Toeplitz matrix B such that:EQUATIONwhere * denotes the 2D convolution operation. Using the properties of matrix multiplication, we can rewrite (7) as:EQUATIONFinally, SEG is formulated as follows:EQUATIONwhere γ seg denotes the guidance scale of SEG. In a straightforward manner, as SEG does not rely on external conditions, it can be used for conditional sampling strategies such as CFG [14] and ControlNet [51] . For the combinatorial sampling with CFG, following [17] , we simply extend (9) for improved conditional sampling with both SEG and CFG as follows:EQUATIONwhich is an intuitive result, as the update rule moves x towards the conditional prediction while keeping it far from the prediction with blurred attention weights.We are likely to get a result with saturation when using a large guidance scale, such as with classifierfree guidance (CFG) [14] , self-attention guidance (SAG) [17] , and perturbed attention guidance (PAG) [1] . This is a significant caveat since we need to increase the scale to achieve a maximum effect with these methods. Contrary to this, we can fix the scale of SEG as justified in Sec. 5.5 and control its maximum effect through σ of the Gaussian blur, making the choice more flexible. For σ, two extreme cases are recognized. If σ → 0, the blurred attention weights remain the same as the original, while when σ → ∞, the attention weights merely adopt a single mean value across spatial axes. We find that even the latter extreme case results in a high-quality outcome, corroborating that we can control the quality to the limit without saturation.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: The paper discuss potential societal impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion on related work: 
 Classifier-free guidance (CFG) [14] , first proposed as a replacement for classifier guidance (CG) [8] is controlled by a scale parameter. The higher we set classifier-free guidance, the more we get faithful, high-quality images. However, it requires external labels, such as text [30] or class [8] labels, making it impossible to apply to unconditional diffusion models. Also, it requires specific traning procedure with label dropping and it is known that high CFG causes saturation [42] .Figure 4 : Conditional generation using ControlNet [51] and SEG. Table 1 : Quantitative comparison of SEG with vanilla SDXL [35] , SAG [17] , and PAG [1] for unconditional generation.Vanilla SDXL [35] Tackling the caveats of CFG, unconditional approaches such as self-attention guidance (SAG) [17] and perturbed attention guidance (PAG) [1] have been proposed. SAG selectively blurs images with the mask obtained from the attention map and guides the generation process given the prediction. This indirect approach causes saturation and noisy images when given a large guidance scale, leading to the selection of a guidance scale less than or equal to 1. PAG guides images using prediction with identity attention, where the attention map is an identity matrix. However, the reliance on heuristics to make perturbed predictions results in unintended side effects. As an example of the side effects of replacing the attention map with identity attention, PAG changes the visual structure and color distribution of an image, as evidenced in Figs. 5, 8, and 9.Contrary to these, we control the effect of SEG through the standard deviation of the Gaussian filter, σ. Moreover, while being theory-inspired, SEG is relatively free from unintended effects. In the following section, we corroborate our claim with extensive experiments. 
implementation details: 
 We build upon the current open-source state-of-the-art diffusion model, Stable Diffusion XL (SDXL) [35] , as our baseline, and do not change the configuration. To sample with SEG, we choose the same attention layers (mid-blocks) and guidance scale as PAG [1] . For SEG and PAG sampling, we use the Euler discrete scheduler [21] , while for SAG [17] , we instead use the DDIM scheduler [45] since the current implementation of SAG does not support the Euler discrete sampler.For SAG and PAG, we use the same configurations they used in the experiments with the previous version of Stable Diffusion, with guidance scales of 1.0 and 3.0, respectively. We set γ seg to 3.0, except in the ablation study. We use various metrics to evaluate quality (FID [10] and CLIP score [37] , calculated with 30k references from the MS-COCO 2014 validation set [28] ) and to assess the extent of change due to applied guidance (LPIPS vgg, alex [52] ). The latter metric, calculated using the outputs of vanilla SDXL, measures the extent of side effects by comparing guided images to their unguided counterparts.In this section, our aim is to demonstrate that with SEG, we can sample plausible images using vanilla SDXL [35] under various conditions and even without any conditions, as demonstrated in Fig. 1 . Furthermore, without the risk of saturation, we can control the quality and plausibility of the samples.For the results, we use σ ∈ {1, 2, 5, 10}. Additionally, as mentioned in Sec. 3.3, we present two extreme cases, σ → 0 (vanilla SDXL) and σ → ∞ (uniform queries).Unconditional generation In this section, our aim is to demonstrate that with SEG, we can sample plausible images from the unconditional mode of the vanilla SDXL, which was originally trained on a large-scale text-to-image dataset. The results are presented in Fig. 1 , Fig. 2 , and Table 1 . The results show a clear tendency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms.In Fig. 2 and Fig. 13 , we show the effectiveness of generating more plausible images, while vanilla SDXL is unable to generate high-quality images without any conditions. The results show a clear tendency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms. When σ is larger, the definition and expression of the samples improve, as the difference in curvature becomes more pronounced.Conditional generation In Figs. 3, 4, 10, 11 , and 14, we display sampling results conditioned on text, Canny, and depth map. Using text (Fig. 3 ), the vanilla SDXL without CFG is unable to generate high-quality images and produces noisy results. Canny and depth map conditioning on SDXL (Fig. 4 , 10, and 11) is achieved through ControlNet [51] , trained on such maps. The results show that SEG enhances the quality and fidelity of the generated images while preserving the textual and structural information provided by the conditioning inputs. Notably, as σ increases, the generated images exhibit improved definition and quality without introducing significant artifacts or deviations from the original condition. The combination with higher CFG scales is shown in Figs. 15 16 17 18 19 .In Table 2 , we show the quantitative results for text-conditional generation in terms of σ. We observe a clear trade-off between image quality (represented by FID and CLIP score) and the deviation from the original sample (represented by LPIPS). We sample 30k images for each σ to compute the metrics.Since the results are visually favorable when we use σ = 10 and σ → ∞, and they are the best in terms of CLIP score and FID, respectively, we adopt those configurations for comparison of unconditional guidance methods. The results are presented in Figs. 5, 8, 9, and Table 1 . Notably, our method achieves better image quality in terms of FID, while remaining similar to the original output of vanilla SDXL as measured by LPIPS, implying a Pareto improvement. In this section, we address two parameters, γ seg and σ, and justify that fixing γ seg is a reasonable choice. In Fig. 6 , we present the results from our testing. The results reveal that increasing γ seg does not generally lead to improved sample quality in terms of FID and CLIP score, due to various issues such as saturation. In contrast, increasing σ tends to improve sample quality and plausibility. This supports the claim that image quality should be controlled by σ, instead of the guidance scale parameter. We sample 30k images for each combination to calculate the metrics. 
conclusion, limitations and societal impacts conclusion: 
 We introduce Smoothed Energy Guidance (SEG), a novel training-and condition-free guidance method for image generation with diffusion models. The key advantages of SEG lie in its flexibility and the theoretical foundation, allowing us to significantly enhance sample quality without side effects by adjusting the standard deviation of the Gaussian filter. We hope our method inspires further research on improving generative models, and extending the approach beyond image generation, for example, to video or natural language processing. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper proposes a guidance method for the current model; therefore, the paper itself poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators or original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets 
 
 Appendix: The paper proposes guidance to enhance quality outcomes. Consequently, the attainable quality of our approach is contingent upon the baseline model employed. Furthermore, the application of SEG to temporal attention mechanisms in video or multi-view diffusion models is not addressed, remaining a promising avenue for future research. It is important to note that the improvements achieved through this method may potentially lead to unintended negative societal consequences by inadvertently amplifying existing stereotypes or harmful biases.A.1 Proof of Lemma 3.1 Let a (i,j) denote the original attention weights and ã(i,j) denote the blurred attention weights, as in the main paper. Assume that the original attention weights are properly padded to maintain consistent statistics. Then, the following shows that the mean of the blurred attention weights remains the same.E i,j [ã (i,j) ] = 1 HW H i=1 W j=1 ã(i,j) = 1 HW H i=1 W j=1 k m=-k k n=-k G(m, n) • a (i+m,j+n) = k m=-k k n=-k G(m, n) •   1 HW H i=1 W j=1 a (i+m,j+n)   = k m=-k k n=-k G(m, n) • E i,j [a (i,j) ] = E i,j [a (i,j) ] • k m=-k k n=-k G(m, n) = E i,j [a (i,j) ]In addition, the variance of the blurred attention weights is smaller than or equal to the variance of the original attention weights.Var i,j [ã (i,j) ] = 1 HW H i=1 W j=1 (ã (i,j) -E i,j [ã (i,j) ]) 2 = 1 HW H i=1 W j=1 k m=-k k n=-k G(m, n) • (a (i+m,j+n) -E i,j [a (i,j) ]) 2 = k m=-k k n=-k k r=-k k s=-k G(m, n) • G(r, s) • Cov[a (i+m,j+n) , a (i+r,j+s) ]Using the Cauchy-Schwarz inequality and the normalization property of the 2D Gaussian filter, we can show that the variance monotonically decreases when we apply Gaussian blur.Var i,j [ã (i,j) ] ≤ k m=-k k n=-k k r=-k k s=-k G(m, n) • G(r, s) • Var[a (i+m,j+n) ] • Var[a (i+r,j+s) ] = k m=-k k n=-k G(m, n) • Var[a (i+m,j+n) ] 2 ≤ k m=-k k n=-k G(m, n) • k m=-k k n=-k G(m, n) • Var[a (i+m,j+n) ] = k m=-k k n=-k G(m, n) • Var[a (i+m,j+n) ] = Var i,j [a (i,j) ]A.2 Proof of Lemma 3.2Applying the second-order Taylor series approximation of e x to our function f around the mean µ, we get:EQUATIONEQUATIONIn the last step, we used the fact thatEQUATIONSince Var[a] > Var[ã], we have:EQUATIONTherefore, the second-order approximation of lse(a) is larger than that of lse(ã).Note that this fact also implies blurring with a Gaussian filter with a bigger variance causes more decrease in the variance of attention weights, because Gaussian filter with a larger variance can always be represented as a convolution of two filters with smaller variances, and the convolution operation is associative.To find the maximum value subject to the constraint a (1,1) + a (1,2) + . . . + a (H,W ) = c for some constant c, we introduce Lagrange multipliers. Let g(a (1,1) , a (1,2) , . . . , a (H,W ) ) = a (1,1) + a (1,2) + . . . + a (H,W ) . The Lagrangian function is defined as:EQUATIONTaking partial derivatives and setting them to zero yields:EQUATIONSolving for a (i,j) , we obtain a (i,j) = ln(λ) for all i = 1, 2, . . . , H and j = 1, 2, . . . , W Summing these equations results in:EQUATIONSubstituting λ back into a (i,j) = ln(λ) gives a (1,1) = a (1,2) = . . . = a (H,W ) = c HW . Therefore, the minimum value of H i=1 W j=1 e a (i,j) is achieved when a (1,1) = a (1,2) = . . . = a (H,W ) .Let a = (a 1 , . . . , a n ) denote the attention values before the softmax operation, and let ã = (ã 1 , . . . , ãn ) denote the attention values after applying the 2D Gaussian blur. Let H denote the Hessian of the original energy, i.e., the derivative of the negative softmax, and H denote the Hessian of the underlying energy associated with the blurred weights.The elements in the i-th row and j-th column of the Hessian matrices are given by:EQUATIONEQUATIONrespectively, where b ij are the elements of the Toeplitz matrix corresponding to the Gaussian blur kernel, and δ ij denotes the Kronecker delta.Assuming ξ(ã) i ξ(ã) j ≈ 0 and ξ(a) i ξ(a) j ≈ 0 for all i and j, which is a reasonable assumption when the number of token is large and the softmax values get small, the non-diagonal elements of the Hessians approximate to 0 and the diagonal elements dominate. Therefore, the determinants of the Hessian matrices are approximated as the product of the dominant terms:EQUATIONWe have the following inequality: n i=1 ξ(ã) i b ii < n i=1 ξ(ã) i = e n j=1 ãj ( n j=1 e ãj ) nwhere the first inequality follows from the property of the Gaussian blur kernel, 0 ≤ b ii < 1, and the second inequality is derived from Lemmas 3.1 and 3.2, which demonstrate the mean-preserving property and the decrease in the lse value when applying a blur. The monotonicity of the logarithm function implies that the denominator involving the blurred attention weights is smaller. Eventually, we obtain the following inequality:EQUATION)This implies that the updated value is derived with attenuated curvature of the energy function underlying the blurred softmax operation compared to that of the original softmax operation.As we previously stated in Section 2.2, we have the dual definition regarding ( 5), where we use swapped indexing. Importantly, the swapped indices can be interpreted as altering the definition of attention weights to A := KQ ⊤ .A similar conclusion can be drawn as in the main paper, except that query blurring becomes key blurring with this definition. To see this, Eq. 7 changes slightly with this definition, using the symmetry of the Toeplitz matrix B:EQUATIONwhere * denotes the 2D convolution operation. Empirically, this altered definition does not introduce a significant difference in the overall image quality, as shown in Fig. 12 .In this section, we present further qualitative results to demonstrate the effectiveness and versatility of our Smoothed Energy Guidance (SEG) method across various generation tasks and in comparison with other approaches.Comparison with previous methods Figs. 8 and 9 provide a qualitative comparison of SEG against vanilla SDXL [35] , Self-Attention Guidance (SAG) [17] , and Perturbed Attention Guidance (PAG) [1] . These comparisons highlight the superior performance of SEG in terms of image quality, coherence, and adherence to the given prompts. SEG consistently produces sharper details, more realistic textures, and better overall composition compared to the other methods.Conditional generation with ControlNet Figs. 10 and 11 showcase the application of SEG in conjunction with ControlNet [51] for conditional image generation. These results illustrate how SEG can enhance the quality and coherence of generated images while maintaining fidelity to the provided control signals. The images demonstrate improved detail, texture, and overall visual appeal compared to standard ControlNet outputs without prompts. Unconditional and text-conditional generation Fig. 13 demonstrates the capability of SEG in unconditional image generation, showcasing its ability to produce high-quality, diverse images without text prompts. Fig. 14 exhibits text-conditional generation results using SEG, illustrating its effectiveness in translating textual descriptions into visually appealing and accurate images.Interaction with classifier-free guidance Figs. 15-19 present a series of experiments exploring the combination of SEG with CFG. In these experiments, the SEG guidance scale (γ seg ) is fixed at 3.0, while the CFG scale is varied. The results demonstrate that SEG consistently improves image quality across different CFG scales without causing saturation or significant changes in the general structure of the images.Ablation study Fig. 20 displays a visual example of unconditional generation with controlled γ seg and σ. Consistent with results in Sec. 5.5, controlling image quality with σ has fewer side effects than controlling with γ seg .The overall pipeline and conceptual framework of SEG are presented in Fig. 7 . Fig. 7 (a) and Fig. 7 (b) depict the original sampling process and the modified sampling process with smoothed energy, respectively. Fig. 7 (c) illustrates the the final prediction (the red arrow) with the guidance scale.Figure 8 : Qualitative comparison of SEG with vanilla SDXL [35] , SAG [17] , and PAG [1] .Figure 9 : Qualitative comparison of SEG with vanilla SDXL [35] , SAG [17] , and PAG [1] . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper received split recommendations after the post-rebuttal discussion. The rebuttal adequately addressed most reviewer concerns about technical correctness and experimental validation. The author-reviewer discussion was engaged, and most remaining doubts were clarified. Overall, the paper presents an interesting technique to improve unconditional sampling from diffusion models that can have a practical impact. Reviewer comments and corrections, particularly those expressed by Reviewer 2oF8, should be carefully included in the final version.

Reviews: review: paper_summary: paper_summary: The method (SEG) discussed in the paper mainly applies an energy-based optimization on the emerging values in the self-attention to reduce the curvature of the energy landscape of attention, leading to improved image quality and less structural change from the original prediction compared to previous approaches. SEG is training- and condition-free and can be used for both unconditional and conditional sampling strategies. The authors validate the effectiveness of SEG by evaluating generated images with and without text conditions, as well as with ControlNet.
strength_weakness: strength_weakness: The quantitative evaluations needed to be presented in a more detailed manner.
The number of works they compared against could be much more.
e.g. imagic (Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I. and Irani, M., 2023. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6007-6017).), LEdit++ (Brack, M., Friedrich, F., Kornmeier, K., Tsaban, L., Schramowski, P., Kersting, K. and Passos, A., 2024. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8861-8870). ) or collaborative diffusion (Huang, Z., Chan, K.C., Jiang, Y. and Liu, Z., 2023. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6080-6090).),
questions: questions: What is the advantage of your method compared to other recent works in text-to-image generation that is mentioned in the weaknesses?
limitations: limitations: The validation can be improved.
paper_summary: paper_summary: The manuscript introduces SEG, a novel training- and condition-free guidance method for enhancing image generation with diffusion models. The method leverages an energy-based perspective of the self-attention mechanism and introduces a technique to reduce the curvature of the energy landscape of attention, thereby improving the quality of generated images. SEG controls the guidance scale by adjusting a Gaussian kernel parameter, offering a flexible and theoretically grounded approach to unconditional and conditional image generation. The authors validate the effectiveness of SEG through extensive experiments on conditions, showcasing its superiority over existing methods like SAG and PAG in terms of sample quality and reduction of unintended effects. My detailed comments are as follows:
strength_weakness: strength_weakness: The method proposed in this manuscript, SEG, relies heavily on the quality of the baseline model and may amplify biases or harmful stereotypes in existing data.
questions: questions: (1) The author proposes that the model can generate images without giving any conditions, so how does the model know what to generate?
(2) In Figure 2, as σ increases, the image becomes clearer, but the style is completely different from the initial one. Why is this? How to determine the optimal value of σ?
(3) What are the main application scenarios of this model? How to generate the image that the user wants without giving any conditions?
(4) In Figure 5, Qualitative comparison of SEG with vanilla SDXL, SAG, and PAG. However, the image is labeled as PEG. Please provide a detailed explanation. Also, the full name of PEG is not given.
(5) In the manuscript, Smoothed Energy Guidance (SEG) is defined multiple times. Generally, abbreviations should be defined the first time they appear and used consistently in subsequent content. Please also check other abbreviations.
limitations: limitations: There is a lack of detailed discussion, such as how to determine the value of σ and why increasing the value of γ does not improve the sample quality in terms of FID and CLIP scores.
paper_summary: paper_summary: This paper presents a method for unconditioned image generation based on Diffusion Models, specifically using Stable Diffusion. The proposed method offers an alternative to classifier-free guidance (CFG), eliminating the need to train a classifier for adding conditions. Traditionally, the CFG denoising function incorporates both conditional and unconditional terms, while unconditional generation only involves an unconditional evaluation of the U-Net. Instead, this method performs unconditional diffusion by introducing an additional term to the traditional reverse process.
This new term is an energy-based component calculated as a proportion of the Gaussian blurring of the self-attention layers in the U-Net. The core idea is to use a blurred version of the unconditional self-attention layers to shift the mean distribution of the original unconditional prediction, thereby "smoothing" the prediction.
The paper employs the standard SDXL model and compares its results against the Perturbed Attention Guidance (PAG) and Self-Attention Guidance (SAG) methods.
strength_weakness: strength_weakness: -  Technical correctness of the paper

The claim in lines 191-192, where the paper asserts that $\sigma \rightarrow 0$ remains the same as the original [image], is not entirely clear, and the intuition behind this is not thoroughly presented. The concern arises because Eq. 7 indicates that the proposed method involves adding another term to the original unconditional generation. According to the literature [10], what this paper refers to as $s_\theta(x,t)$ corresponds to a distribution, as does $\tilde{s_{\theta}}(x,t)$. However, having $\sigma$ approaching to 0 does not imply that this term becomes 0; rather, it means that the Gaussian filter affecting the calculation of the self-attention is 0. Therefore, the inference drawn from this reasoning is that if the filter's variance approaches 0, then $s_\theta(x,t)$ approximates to $\tilde{s}_\theta(x,t)$.

Since this paper relies on a strong theoretical background, it is crucial to provide a thorough demonstration and explanation of all claims. For instance, in line 150, what does it mean to be under a "reasonable" assumption to demonstrate the attenuation of the Gaussian curvature? Additionally, what does the paper refer to when discussing the property of linear mapping in the blurring of the queries between lines 167 and 169?

As the method uses a Gaussian kernel for the blurring process, it involves a sigma parameter derived from the Gaussian filter. However, there is insufficient discussion on how this parameter significantly affects the saturation of the generated image. From line 211, it can be interpreted that sigma controls the saturation, but between lines 274-276, the paper indicates that saturation can be altered by using the guidance scale for this method. The question is how sigma and gamma can be balanced to avoid over-saturated images and whether other characteristics of the image (brightness, darkness, vibrance, etc.) can be controlled through these parameters.

- Experimental validation

This paper limits its exploration by not providing further comparisons using other diffusion models as backbones. This omission raises questions about the proposed method's generalization capability.

It is not clear why the paper does not report the perceptual metric for the Vanilla SDXL in Table 1.
The lack of a metric to compare the perceptual performance of the Vanilla SDXL against the proposed method raises a concern, as the validation is then limited to just a few image examples presented in this manuscript. Without this measure, it is difficult to fully understand the overall performance of the proposed method compared to the baseline and other state-of-the-art methods.

The disclaimer in lines 231–232 regarding the FID is acknowledged. However, when it comes to image generation, these are the metrics that the community has adopted and learned to interpret. Of course, this does not limit new research from proposing new ways to measure image generation to provide a "strict measure of image quality" and other attributes.

This paper claims (line 14-15) that its implementation does not significantly increase computational cost. However, it is unclear how the computational requirements should be adjusted to use it. Moreover, the paper directly compares its method against unconditional generation, which only requires a single network evaluation, while this method (according to Eq. 7) requires at least two network evaluations. Furthermore, using a conditional input would necessitate at least three network evaluations, likely leading to a considerable increase in computational cost. It is suggested that the paper include an analysis of the trade-off in terms of the number of function evaluations (NFEs) to fully understand how many network evaluations are needed and how this method impacts computational cost.

It would have been interesting to see how other filter kernels perform besides Gaussian blurring.

- Presentation 

Although this paper is well-written and presented, its intelligibility could be improved by adding a few more sentences to clarify some of the intuitions described above.
questions: questions: The questions are listed in the explanation of the weaknesses. It is encouraged to provide further explanation to the presented concerns to have the chance of increasing this initial score. 
In general, the paper, while presenting theoretical justifications, needs clearer explanations of its claims, particularly concerning Gaussian curvature attenuation and the linear mapping property in blurring queries. The impact of the sigma parameter on image saturation requires more discussion, including balancing sigma and gamma. The lack of comparisons with other diffusion models limits the assessment of the method's generalization, and the absence of perceptual metrics for Vanilla SDXL restricts validation to a few examples. While the FID disclaimer is noted, traditional community metrics should be used.
limitations: limitations: NA
paper_summary: paper_summary: The paper proposes a technique to improve unconditional sampling from diffusion models. The main idea is to translate the notion of classifier-free guidance (CFG) to the case in which there is no condition available. To this end, the paper notes that the conditional prediction is "sharp", while the unconditional prediction is "smooth", or more simply put, the unconditional prediction is smoother than the conditional one. The paper also notes that applying Gaussian filtering on the attention weights of the model's unconditional prediction yields an (even) smoother prediction. The paper then combines these two observations, and proposes SEG, a version of CFG that is applicable to the unconditional case: CFG requires the conditional and unconditional predictions; SEG replaces the conditional prediction with the unconditional, while replacing the unconditional with its smoothed version. By doing this procedure, SEG manages to preserve the sharpness-smoothness relation that CFG has between the factors (the conditional and unconditional predictions).

The paper then demonstrates that this procedure translates into improved performance compared to reasonable competitors on standard benchmarks.
strength_weakness: strength_weakness: ## Weaknesses that don't affect my rating
- The Method section could strongly benefit from a figure illustrating on what object, exactly, is the Gaussian blur being applied. The paper currently doesn't have a "pipeline" figure
- I understand the value of Lemma 3.1, but I don't think it's necessary to provide a proof in the main paper. As a side note, I don't even think it's necessary to provide a proof: as far as I understand, this fact is widely known, since the Gaussian filter is normalized and symmetric, no?
  - For instance, I think a well-known intuition in the computer vision community is that (infinite) successive applications of Gaussian blurring on an image result in an image with a single color, that corresponds to the average color of the initial image (i.e. same mean, and zero variance).
- L188 mentions PAG. It would be useful to remind the reader what it stands for. (I think it was originally just mentioned in the introduction)
- Unless I'm misunderstanding, L211 claims that, contrary to two other methods--SAG and PAG--the presented method is both training- and condition-free. I'm not sure I follow: aren't those two methods both also training- and condition-free?
- I understand what the authors mean in L231 by "FID has been found not to be a strict measure of image quality". However, FID in itself does not measure image quality, but distance w.r.t. some distribution
------------
## Weaknesses that don't affect my rating, but should be addressed:
- Eq. (5): please consider explaining that "lse" standards for logsumexp a bit before L85: by then it's a bit confusing already
- Isn't L86 redundant (since the same statement is made--and referenced--in L75?)
- L95: please consider denoting the LSE with some convention: just writing "lse" lends itself to confusion
- L97: a negative prediction of what?
- L108: to make which operation tractable?
- L124: increases the LSE or decreases (according to L127). My intuition is that it decreases, since LSE approximates the maximum of the tensor
- The proof of proposition 3.1 could be made shorter, or sent to the appendix
- L177: "Note that the Gaussian blur can also be applied to K" I presume this is true because of the commutation property of the convolution operation. Perhaps explicitly stating that in the paper would be better

Typos and such:
- L9: actually?
- L39: why "actually"?
- L70: conditioning on
- L72: demonstrates its prevalence? sounds weird
- L111 and equation at L 113: boldface "a"?
- Minor comment for all over the paper: use \eqref for equations
- L230: "how much guided images are altered from unguided ones" sounds a bit weird
- L232: "being more favorable to users" sounds weird
- Fig. 2 and 11: I think there's a repeated sample? (the one indoors)
questions: questions: - I can understand that the main interest of the paper was improving unconditional generation. The paper achieves this objective by, in a sense, computing a less sharp/more smooth prediction than the unconditional one; this prediction is combined with the original unconditional prediction in the same way that CFG operates. I think that approach is interesting from the point of view of efficiency: a useful prediction is being obtained by an inexpensive operation (i.e. smoothing), instead of another forward pass. Given this context, doesn't it make sense to rather aim at improving conditional generation? (which is arguably the most important one). I think the paper's proposal could be used directly to improve the efficiency conditional generation with CFG: do the conditional forward passes, and replace the unconditional ones with the ones that result from Gaussian smoothing (that is, I think you can replace an entire function evaluation with simple Gaussian smoothing of the conditional prediction). Have the authors considered this option? 
[I think Eq. (8) with gamma_seg=0 is somewhat similar, but I'd be thinking of replacing s(x,t) with a \tilde{s}(x,t) = smooth(s(x,t,c))]
limitations: limitations: Yes, they have
paper_summary: paper_summary: The paper presents a condition-free guidance method for diffusion models. The guidance is generated from the self-attention mechanism to perform guidance from an energy-based perspective as an alternative to classifier-free guidance. With this, the work aims to train the models for improved quality performance in conditional and unconditional image generation.
strength_weakness: strength_weakness: As a suggestion by the paper is to refer to a qualitative assessment, what would be the criteria to consider a model or ablation generates a better quality?
questions: questions: Questions were mentioned in the weaknesses section. Additionally, it would be recommended to introduce the relationship between reducing the curvature of the enemy function and guidance for context.
limitations: limitations: The paper discusses potential limitations and societal impact of generative models in the conclusion section.

Comments: title: Rensponse to Rebuttal
Response to the Final Questions
Final Questions
Further Questions Welcome
Further Questions Welcome
Further Questions Welcome
Further Questions Welcome
Further Questions Welcome
comment: Thank you for the author's response. I will consider them in my final rating.
Thank you for your valuable feedback. We appreciate your suggestions, but we believe there are some misunderstandings that we'd like to address.

1. Regarding comparisons with state-of-the-art methods, **we have already included experiments using ControlNet in our main paper (Figures 4, 9, and 10)**. These comparisons directly address your concern about contextualizing our contributions against powerful approaches.

2. In addition, we deliberately did not compare our method with **text-based image editing** approaches like LEdit++ because they do not align with our research goals. Our focus is on different editing paradigms, making such comparisons inappropriate as baselines for our work. This point is also addressed in the rebuttal.

As we are at the end of the reviewer-author discussion period, we respectfully ask that you consider the information already provided in our paper and previous responses. We have addressed many of these points earlier and believe our work stands on its own merits within its intended scope. We hope this clarifies our position and demonstrates that we have indeed addressed many of your concerns within the constraints of our research focus. We urge you to reconsider your evaluation in light of this information.

Again, thank you very much for your thoughtful and valuable questions.

Best regards,

Authors of Submission #4721
I thank the authors who have provided additional quantitative evaluation using FD-DINOv2 scores; the comparison is still limited. It may be worth suggesting they expand their quantitative comparisons to include more baseline methods and metrics. 
The highlights the authors provided regarding the generality of their approach are very interesting; it would be beneficial to see more direct comparisons with state-of-the-art methods in both conditional and unconditional settings to better contextualize their contributions. Methods like LEdit++ and ControlNet are powerful and intuitive approaches, and if the proposed method holds the generalizability advantage, it can be applied along the SOTA methods and improved.
I thank the authors for the response. I have decided to maintain my score.
We appreciate your valuable suggestions and feedback. Here, we provide further explanations to address your questions.

Basically, our main theoretical results do not only apply to SDXL but also to different backbones across various conditions equipped with self-attention mechanisms. In addition, we report experiments using the conditional EDM2-S model [A] trained on ImageNet-64. Fig. 3 in the attached PDF showcases uncurated qualitative samples from this model. We also present FD-DINOv2 scores calculated using 50k samples to assess fidelity to the real image data, which is well-aligned with human perception [A]. These results demonstrate that our method generalizes well to a different backbone, showing significant improvements in sample quality.
| Method     | FD-DINOv2  |
|------------|------------|
| No guidance| 95.1915    |
| SEG        | **47.4733**|

[A] Karras et al. "Analyzing and improving the training dynamics of diffusion models." CVPR 2024.

Additionally, as per the reviewer's suggestion, we conducted experiments using various filter kernels and calculated the scores using 3k samples with the same random seeds for the EDM2-S model:
| Filter type | Identity | Bilateral ($\sigma_1 = 1$, $\sigma_2 = 0.5$) | Laplacian | Gaussian ($\sigma = 1$) | Gaussian ($\sigma = 10$) | Gaussian ($\sigma \to \infty$) |
|-------------|-----------|---------|-------------------|--------------------------------|--------------------------------|---------------------------------------------------------------------|
| FD-DINOv2   | 230.676   | 226.757                                           | 195.178           | 200.623                        | 190.384                        | **190.089**  

$\sigma_1$ and $\sigma_2$ used in the bilateral filter denote the parameters controlling the spatial extent of the filter and the influence of intensity differences, respectively.

In this experiment, SEG using the Gaussian filter outperforms those using different filters. While the other filters enhance the score, SEG is the approach that benefits from our theoretical grounding, allowing us to gradually reduce the energy curvature by increasing a single parameter, $\sigma$, which is the main contribution of our paper.

Thank you again for your thoughtful and valuable questions. We hope this clarification addresses your remaining concerns.
Dear Authors,

Thank you for your response! The initial comments have clarified the concerns regarding Gaussian curvature attenuation, the linear mapping property in blurring queries, and metrics validation. However, I am still curious about whether this methodology generalizes to other diffusion backbones. Additionally, could you please provide some insight into how other filter kernels perform compared to Gaussian blurring?

Best,
Reviewer 9qC6
Dear Reviewer cNNp,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721
Dear Reviewer CTSN,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721
Dear Reviewer 9qC6,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721
Dear Reviewer Rhhf,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721
Dear Reviewer 2oF8,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721