Understanding and Improving Training-free Loss-based Diffusion Guidance

Authors: yifei  shen
xinyang  jiang
yifan  yang
yezhen  wang
dongqi  han
dongsheng  li
microsoft  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 11:04:58

Abstract: Adding additional guidance to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free loss-based guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of training-free guidance, as well as overcome its limitations. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free guidance is more susceptible to misaligned gradients and exhibits slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.

Main paper: 
 introduction: 
 Diffusion models represent a class of powerful deep generative models that have recently broken the long-standing dominance of generative adversarial networks (GANs) [8] . These models have demonstrated remarkable success in a variety of domains, including the generation of images and videos in computer vision [33, 3] , the synthesis of molecules and proteins in computational biology [20, 52] , as well as the creation of trajectories and actions in the field of reinforcement learning (RL) [21] .One critical area of research in the field of diffusion models involves enhancing controllability, such as pose manipulation in image diffusion [50] , modulation of quantum properties in molecule diffusion [20] , and direction of goal-oriented actions in RL diffusion [23] . The predominant techniques for exerting control over diffusion models include classifier guidance and classifier-free guidance. Classifier guidance involves training a time-dependent classifier to map a noisy image, denoted as x t , to a specific condition y, and then employing the classifier's gradient to influence each step of the diffusion process [8] . Conversely, classifier-free guidance bypasses the need for a classifier by training an additional diffusion model conditioned on y [18] . However, both approaches necessitate extra training to integrate the conditions. Moreover, their efficacy is often constrained when the data-condition pairs are limited and typically lack the zero-shot generalization capability.Recently, several studies [2, 49, 37] have introduced training-free guidance that builds upon the concept of classifier guidance. These models eschew the need for training a classifier on noisy images; instead, they estimate the clean image from its noisy counterpart using Tweedie's formula and then employ pretrained networks, designed for clean images, to guide the diffusion process. Given that checkpoints for these networks pretrained on clean images are widely accessible online, this form of guidance can be executed in a zero-shot manner. A unique advantage of training-free guidance is that it can be applied to universal control formats, such as style, layout, and FaceID [2, 49, 37] without any additional training efforts. Furthermore, these algorithms have been successfully applied to offline reinforcement learning, enabling agents to achieve novel goals not previously encountered during training. In contrast to classifier guidance and classifier-free guidance, it is proved in Appendix E of [27] that training-free guidance does not offer an approximation to the exact conditional energy. Therefore, from a theoretical perspective, it is intriguing to understand how and when these methods succeed or fail. From an empirical standpoint, it is crucial to develop algorithms that can address and overcome these limitations. This paper seeks to deepen the understanding of training-free guidance by examining its mechanisms and inherent limitations, as well as overcoming these limitations. Specifically, our major contributions can be summarized as follows:• How does training-free guidance work? Although exact conditional energy is difficult to approximate in a training-free manner, from the optimization standpoint, we show that training-free guidance can effectively decrease the guidance loss function. The optimization perspective clarifies the mystery of why the guidance weights should be meticulously designed in relation to the guidance function and diffusion time, as observed in [49] . • When does training-free guidance not work? We theoretically identify the susceptibility of training-free guidance to misaligned gradient issues and slower convergence rates. We attribute these challenges to a decrease in the smoothness of the guidance network in contrast to the classifier guidance. • Improving training-free guidance: We introduce random augmentation to alleviate the misaligned gradient and Polyak step size scheduling to improve convergence. The efficacy of these methods is empirically confirmed across various diffusion models (i.e., image diffusion and motion diffusion) and under multiple conditions (i.e., segmentation, sketch, text, object avoidance, and targeting) 2 . 
diffusion models: 
 Diffusion models are characterized by forward and reverse processes. The forward process, occurring over a time interval from 0 to T , incrementally transforms an image into Gaussian noise. On the contrary, the reverse process, from T back to 0, reconstructs the image from the noise. Let x t represent the state of the data point at time t; the forward process systematically introduces noise to the data by following a predefined noise schedule given by x t = √ α t x 0 + σ t ϵ t , where α t ∈ [0, 1] is monotonically decreasing with t, σ t = √ 1 -α t , and ϵ t ∼ N (0, I) is random noise. Diffusion models use a neural network to learn the noise at each step:min θ E xt,ϵ,t [∥ϵ θ (x t , t) -ϵ t ∥ 2 2 ] = min θ E xt,ϵ,t [∥ϵ θ (x t , t) + σ t ∇ xt log p t (x t )∥ 2 2 ],where p t (x t ) is the distribution of x t . The reverse process is obtained by the following ODE:EQUATIONwheref (t) = d log √ αt dt , g 2 (t) = dσ 2 t dt -2 d log √ αt dt σ 2 t .The reverse process enables generation as it converts a Gaussian noise into the image.In diffusion control, we aim to sample x 0 given a condition y. The conditional score function is expressed as follows:EQUATIONThe conditions are specified by the output of a neural network, and the energy is quantified by the corresponding loss function. If ℓ(f ϕ (•), •) represents the loss function as computed by neural networks, then the distribution of the clean data is expected to follow the following formula [2, 49, 37] :EQUATIONFor instance, consider a scenario where the condition is the object location. In this case, f ϕ represents a fastRCNN architecture, and ℓ denotes the classification loss and bounding box loss. By following the computations outlined in [27] , we can derive the exact formula for the second term in the RHS of (2) as:EQUATIONClassifier guidance [8] Training-free loss-based guidance [2, 49, 6] puts the expectation in (4) inside the loss function:EQUATIONwhere (a) uses Tweedie's formulaE p(x0|xt) (x 0 ) = xt-σtϵ θ (xt,t) √ αt. Leveraging this formula permits the use of a pretrained off-the-shelf network designed for processing clean data. The gradient of the last term in the energy function is obtained via backpropagation through both the guidance network and the diffusion backbone. 
how does training-free guidance work?: 
 On the difficulty of approximating ∇ xt log p t (y|x t ) in high-dimensional space. Despite being intuitive, [27] has shown that training-free guidance in (5) does not offer an approximation to the true energy in (4) . The authors of [37] consider to directly approximate (4) with a Gaussian distribution:EQUATIONwhere q(x 0 |x t ) is chosen as N (E p(x0|xt) (x 0 ), r 2 t I) and r t is a tunable parameter. As demonstrated in [37] , the approximation is effective for one-dimensional distribution. However, we find that the approximation denoted by (a) does not extend to high-dimensional data (e.g., images) if the surrogate distribution q is sub-Gaussian. This is due to the well-known high-dimensional probability phenomenon [43] that if q has sub-Gaussian coordinates (e.g., iid and bounded), then q(x 0 |x t ) tends to concentrate on a spherical shell centered at E p(x0|xt) (x 0 ) with radius r t (details are in Appendix C.1). Since the spherical shell represents a low-dimensional manifold with zero measure in the high-dimensional space, there is a significant likelihood that the supports of p(x 0 |x t ) and q(x 0 |x t ) do not overlap, rendering the approximation (a) ineffective.Understanding training-free guidance from an optimization perspective. We instead analyze the training-free guidance from the optimization perspective. Intuitively, in each step, the gradient is taken and the loss of the guidance network decreases. At the initial stage of the diffusion (t is large), the diffusion trajectory can exhibit substantial deviations between adjacent steps and may increase the objective value. So the objective value will oscillate at the beginning. When t is smaller, the change to the sample is more fine-grained, leading to a bounded change in the objective value. Therefore, the objective value is guaranteed to decrease when t is small, as showing in the next proposition. Proposition 3.1. Assume that the guidance loss function ℓ(f ϕ (x 0 ), y) is µ-PL (defined in Defintion D.2 in appendix) and L f -Lipschitz with respect to clean images x 0 , and the score function , y and denote∇ log p t (x t ) is L p -Lipschitz (κ 1 = µλ 2 min L f (1+Lp) √ αtσ 4 t . After one gradient step xt = x t -η∇ xt ℓ t (x t ), η = √ αt L f (1+Lp) , we have ℓ t ( xt ) ≤ (1 -κ 1 )ℓ t (x t );(2) Consider a diffusion process that adheres to a bounded change in the objective function such that for any diffusion step, i.e., ℓ t-1 (x t-1 ) ≤ ℓt ( xt) (1-κ2) for some κ 2 < κ 1 , then the objective function converges at a linear rate, i.e., ℓ t-1 (x t-1 ) ≤ 1-κ1 1-κ2 ℓ t (x t ). The proof is given in Appendix D.2. The Lipschitz continuity and PL conditions are basic assumptions in optimization, and it has been shown that neural networks can locally satisfy these conditions [5] . These assumptions, while not essential, simplify the proof and results for clarity, similar to the assumptions taken in [26] . The optimization perspective clarifies the mystery of why the guidance weights (i.e., the step size in optimization) should be carefully selected with respect to the guidance function and time t. For example, in [49] , most guidance weights η are chosen to be proportional to √ α t and dependent on guidance network, which differs from the weights used in classifier guidance and aligns with our step size in Proposition 3.1.Then we empirically verify Proposition 3.1 via experiments in Figure 1 . We use ResNet-50 trained on clean images to guide ImageNet pretrained diffusion models. The loss value at each diffusion step is plotted. As a reference, we choose 100 images from the class "indigo bird" in ImageNet training set and compute the loss value, which is referred to as "Correct Image Loss" in the figure. The objective value oscillates when t is large, followed by a swift decrease, which verifies our analysis. More convergence figures are given in Figure 7 in Appendix.An intriguing aspect of the theory is that the loss remains low regardless of the success of the guidance, akin to the loss associated with correct images. Figure 1b demonstrates this phenomenon: despite the absence of an indigo bird in the image, the loss is still minimal. This phenomenon can be attributed to the effect of misaligned gradients, which is explored in detail in the following subsection.In this subsection, we examine the disadvantages of employing training-free guidance networks as opposed to training-based classifier guidance.Training-free guidance is more sensitive to the misaligned gradient. Adversarial gradient is a significant challenge for neural networks, which refers to minimal perturbations deliberately applied to inputs that can induce disproportionate alterations in the model's output [38] . The resilience of a model to adversarial gradients is often analyzed through the lens of its Lipschitz constant [34] . If the model has a lower Lipschitz constant, then the output is less sensitive to the input perturbations and thus is more robust.In the classifier or training-free guidance, the gradient of the guidance network is added to the image. In contrast to yielding a direction that meaningfully minimizes the loss, the adversarial gradient primarily serves to minimize the loss in a manner that is not necessarily aligned with the intended guidance direction. As a result, we refer the adversarial gradient of guidance network as the misaligned gradient in diffusion guidance. The proof is given in Appendix D.3. We then support Proposition 3.2 with both qualitative and quantitative experiments. For qualitative experiments, we present visualizations of the accumulated gradients for both the time-dependent and off-the-shelf time-independent classifiers corresponding to different classes in Figure 2b and Figure 2c , respectively. These visualizations are generated by initializing an image with a random background and computing 1000 gradient steps for each classifier. For the time-dependent classifier, the input time for the t-th gradient step is 1000 -t. The images are generated purely by the classifier gradients without diffusion involved. For comparative analysis, we include the accumulated gradient of an adversarially robust classifier [35] , as shown in Figure 2a , which has been specifically trained to resist misaligned (adversarial) gradients. The resulting plots reveal a stark contrast: the gradient of the time-dependent classifier visually resembles the target image, whereas the gradient of the time-independent classifier does not exhibit such recognizability. This observation suggests that off-the-shelf time-independent classifiers are prone to generating misaligned gradients for guidance compared to the time-dependent classifier used in classifier guidance. The quantitative experiments are given in Table 4 in Appendix.Figure 2 provides a more intuitive visual explanation of diffusion guidance compared to the existing formula-based approaches as shown in (7) . The gradient produced by the guidance network represents a valid image. When these gradients are incorporated into the images, the diffusion model is able to identify the object and enhance it into a clearer and more vivid representation.Training-free guidance slows down the convergence of reverse ODE. The efficiency of an algorithm in solving reverse ordinary differential equations is often gauged by the number of non-linear function estimations (NFEs) required to achieve convergence. This metric is vital for algorithmic design, as it directly relates to computational cost and time efficiency [36] . In light of this, we explore the convergence rates associated with various guidance paradigms, beginning our analysis with a reverse ODE framework that incorporates a generic gradient guidance term. The formula is expressed asEQUATIONwhere h(•, •) can be either a time-dependent classifier or a time-independent classifier with Tweedie's formula. The subsequent proposition elucidates the relationship between the discretization error and the smoothness of the guidance function.Proposition 3.3. Let u(x t , t) = ϵ θ (x t , t) + ∇ xt v(x t , t) in (7), h max = max t 1 2 log( αt 1-αt ) -log( αt-1 1-αt-1) . Assume we run DDIM solver for M steps andM = O(1/h max ). Then the error is bounded by O((1 + L M )/M ).The proof is given in Appendix D.4. Proposition 3.2 establishes that time-dependent classifiers exhibit superior gradient Lipschitz constants compared to their off-the-shelf time-independent counterparts. This disparity in smoothness slows down the convergence for training-free guidance methods, necessitating a greater number of NFEs to achieve the desired level of accuracy when compared to classifier guidance. To provide quantitative support, we compare the convergence speed of training-based PPAP [12] and training-free FreeDoM in Table 5 in Appendix. 
improving training-free guidance: 
 In this section, we propose to adopt random augmentation to mitigate the misaligned gradient issue, and Polyak step size [15] to mitigate the convergence issue. In addition to these two techniques, our method and baselines will also incorporate a trick named time travel, often referred to as "restart sampling", and its theoretical framework is detailed in [46] .Algorithm 1 Random Augmentation for t = T, • • • , 0 do x t-1 = DDIM(x t ) x0 = xt-σtϵ θ (xt,t) √ αt ▷ Tweedie's formula g t = 1 |T | T ∈T ∇ xt ℓ(f ϕ (T ( x0 )), y) x t-1 = x t-1 -η • g t end for Algorithm 2 Polyak Step Size for t = T, • • • , 0 do x t-1 = DDIM(x t ) x0 = xt-σtϵ θ (xt,t) √ αt ▷ Tweedie's formula g t = ∇ xt ℓ(f ϕ ( x0 ), y) x t-1 = x t-1 -η • ∥ϵ θ (xt,t)∥ ∥gt∥ 2 2 • g t end forAs established by Proposition 3.2, the introduction of Gaussian perturbations enhances the Lipschitz property of a neural network. A direct application of this principle involves creating multiple noisy instances of an estimated clean image and passing them into the guidance network, a method analogous to the one described in (6) . However, given the high-dimensional nature of image data, achieving a satisfactory approximation of the expected value necessitates an impractically large number of noisy copies. To circumvent this issue, we propose an alternative strategy that employs a diverse set of data augmentations in place of solely adding Gaussian noise. This approach effectively introduces perturbations within a lower-dimensional latent space, thus requiring fewer samples. The suite of data augmentations utilized, denoted by T , is derived from the differentiable data augmentation techniques outlined in [51] , which encompasses transformations such as translation, resizing, color adjustments, and cutout operations. The details are shown in Algorithm 1 and the rationale is shown in the following proposition. Proposition 4.1. (Random augmentation improves smoothness) Given a bounded non-Lipschitz loss function ℓ(x), the loss l(x) = E ϵ∼p(ϵ) [ℓ(x + ϵ)] is C R n ∥∇p(t)∥ 2 dt-Lipschitz and its gradient is C R n ∥∇ 2 p(t)∥ op dt-Lipschitz.The proof is shown in Appendix D.5. Echoing the experimental methodology delineated in Section 3.2, we present an analysis of the accumulated gradient effects when applying random augmentation to a ResNet-50 model. Specifically, we utilize a set of |T | = 10 diverse transformations as our augmentation strategy. The results of this experiment are visualized in Figure 2d , where the target object's color and shape emerge in the gradient profile. This observation suggests that the implementation of random augmentation can alleviate the misaligned gradient issue. The quantitative effect of random augmentation is given in Table 4 in Appendix. The computational efficiency of random augmentation is further discussed in Appendix C.4.In Section 3.1, we analyzed training-free guidance from the optimization perspective. To accelerate the convergence, gradient step size should be adaptive to the gradient landscape. We adopt Polyak step size, which has near-optimal convergence rates under various conditions [15] . The algorithm is shown in Algorithm 2 and the term ∥ϵ θ (x t , t)∥ is used to both estimate the gap to optimal values and balance the magnitude of diffusion term and guidance term.We implement Polyak step size within the context of a training-free guidance framework called FreeDoM [49] and benchmark the performance of this implementation using the DDIM sampler with 50 steps. As shown in Figure 3 , FreeDoM is unable to effectively guide the generation process when faced with a significant discrepancy between the unconditional generation and the specified condition. An illustrative example is the difficulty in guiding the model to generate faces oriented to the left when the unconditionally generated faces predominantly orient to the right, as shown in Figure 3b . This challenge, which arises due to the insufficiency of 50 steps for convergence under the condition, is ameliorated by substituting gradient descent with adaptive step size, thereby illustrating the benefits of employing a better step size in the guidance process. The quantitative experiments are given in Table 5 in Appendix. 
experiments: 
 In this section, we evaluate the efficacy of our proposed techniques across various diffusion models and guidance conditions. We compare our methods with established baselines: Universal Guidance (UG) [2] , Loss-Guided Diffusion with Monte Carlo (LGD-MC) [37] , Training-Free Energy-Guided Diffusion Models (FreeDoM) [49] , and Manifold Preserving Guided Diffusion (MPGD) [16] .LGD-MC utilizes (6) while UG and FreeDoM are built on (5) . MPGD utilizes an auto-encoder to ensure the manifold constraints. Furthermore, time travel trick (Algorithm 3) is adopted in UG, FreeDoM, and MPGD to improve the performance. Please refer to Appendix B for details of baselines. For the sampling method, DDIM with 100 steps is adopted as in [49, 37] . The method "Ours" is built on FreeDoM, with Polyak step size and random augmentation.In this subsection, we adopt the experimental setup from [49] . Specifically, we utilize the CelebA-HQ diffusion model [19] to generate high-quality facial images. We explore three guidance conditions: segmentation, sketch, and text. For segmentation guidance, BiSeNet [48] generates the facial segmentation maps, with an ℓ 2 -loss applied between the estimated map of the synthesized image and the provided map. Sketch guidance involves using the method from [45] to produce facial sketches, where the loss function is the ℓ 2 -loss between the estimated sketch of x0 and the given sketch. For text guidance, we employ CLIP [32] as both the image and text encoders, setting the loss to be the ℓ 2 distance between the image and text embeddings.We randomly select 1000 samples each of segmentation maps, sketches, and text descriptions. The comparative results are presented in Table 1 . Consistent with [49] , the time-travel number for all methods is set to s = 1. Figure 4 displays a random selection of the generated images. More image samples are provided in the supplementary materials. We find that the baselines failed to guide if the condition differs from unconditionally generated images significantly, as discussed in Section 4.2.For the unconditional ImageNet diffusion, we employ text guidance in line with the approach used in FreeDoM and UG [2, 49] . We utilize CLIP-B/16 as the image and text encoder, with cosine similarity serving as the loss function to measure the congruence between the image and text embeddings.To evaluate performance and mitigate the potential for high-scoring adversarial images, we use CLIP-L/14 for computing the CLIP score. In FreeDoM and MPGD-Z, resampling is conducted for time steps ranging from 800 to 300, with the time-travel number fixed at 10, as described in [49] .Given that UG resamples at every step, we adjust its time-travel number s = 5 to align the execution time with that of FreeDoM. The textual prompts for our experiments are sourced from [25] . The comparison of different methods is depicted in Table 2 . The corresponding randomly selected images are illustrated in Figure 5 . The table indicates that our method achieves the highest consistency with the provided prompts. As shown in Figure 5 , LGD-MC and MPGD tend to overlook elements of the prompts. Both UG and FreeDoM occasionally produce poorly shaped objects, likely influenced by misaligned gradients. Our approach addresses this issue through the implementation of random augmentation. Additionally, none of the methods successfully generate images that accurately adhere to positional prompts such as "left to" or "below". This limitation is inherent to CLIP and extends to all text-to-image generative models [41] . More image samples are provided in the supplementary materials.In this subsection, we extend our evaluation to human motion generation using the Motion Diffusion Model (MDM) [40] , which represents motion through a sequence of joint coordinates and is trained on a large corpus of text-motion pairs with classifier-free guidance. We apply the targeting guidance and object avoidance guidance as described in [37] . Let x 0 (t) denote the joint coordinates at time t, y t the target location, y obs the obstacle location, r the radius of the objects, and T the total number of frames. The loss function is defined as follows:EQUATIONOur experimental configuration adheres to the guidelines set forth in [37] . We assess the methods using the targeting loss (the first term in (8)), the object avoidance loss (the second term in (8)), and the CLIP score calculated by MotionCLIP [39] . In this application, MPGD-Z cannot be applied as there are no auto-encoder. MPGD w/o proj suffers from the shortcut and cannot achieve good performance, as discussed in Appendix B.2. In our method, random augmentation is omitted because the guidance is not computed by neural networks so the adversarial gradient issues are not obvious.The quantitative results of our investigation are summarized in Table 3 , while Figure 6 showcases randomly selected samples. Our methods exhibit enhanced control quality over the generated motion.The videos are provided in the supplementary materials. Loss is reported as a two-component metric: the first part is the MSE between the target and the actual final position of the individual; the second part measures the object avoidance loss.Due to space limitation, we only introduce the related work on training-free guidance while leaving more related work in Appendix A. The current training-free guidance strategies for diffusion models can be divided into two primary categories. The first category is the loss-based guidance in this paper, which is universally applicable to universal control formats and diffusion models. These methods predict a clean image, subsequently leveraging pretrained networks to guide the diffusion process. Central to this approach are the algorithms based on (5), which have been augmented through techniques like time-travel [2, 49] and the introduction of Gaussian noise [37] . The adjoint sensitivity method [31] and spherical Gaussian constraint [47] have been adopted to estimate a more accurate guidance gradient. Extensions of these algorithms have found utility in domains with constrained data-condition pairs, such as molecule generation [14] , and in scenarios necessitating zero-shot guidance, like open-ended goals in offline reinforcement learning [44] . In molecular generation and offline reinforcement learning, they outperform training-based alternatives as additional training presents challenges. This paper delves deeper into the mechanics of this paradigm and introduces a suite of enhancements to bolster its performance. The efficacy of our proposed modifications is demonstrated across image and motion generation, with promising potential for generalization to molecular modeling and reinforcement learning tasks.The second category of training-free guidance is tailored to text-to-image or text-to-video diffusion models, which is based on insights into their internal backbone architecture. For instance, object layout and shape have been linked to the cross-attention mechanisms [17] , while network activations have been shown to preserve object appearance [42] . These understandings facilitate targeted editing of object layout and appearance (Diffusion Self-Guidance [11] ) and enable the imposition of conditions in ControlNet through training-free means (FreeControl [30] ). Analyzing these methodologies is challenging due to their reliance on emergent representations during training. Nonetheless, certain principles from this paper remain relevant; for example, as noted in Proposition 3.3, these methods often necessitate extensive diffusion steps, with instances such as [30, 11] employing 1000 steps. A thorough examination and refinement of these techniques remain an avenue for future research. 
conclusions: 
 In this paper, we conducted a comprehensive investigation into training-free guidance, which employs pretrained diffusion models and guides them using the off-the-shelf trained on clean images. Our exploration delved into the underlying mechanisms and fundamental limits of these models. Moreover, we proposed a set of enhancement techniques and verified their effectiveness both theoretically and empirically.Limitations. Despite our efforts to mitigate the shortcomings of training-free methods and enhance their performance, certain limitations remain. Notably, the refined training-free guidance still necessitates a higher number of NFEs when compared with extensive training methods such as classifier-free guidance. This is because misaligned gradient cannot be fully eliminated without training.Ethical Consideration. Similar to other models designed for image creation, our model also has the unfortunate potential to be used for creating deceitful or damaging material. We pledge to restrict the usage of our model exclusively to the realm of research to prevent such misuse. LGD-MC A Related WorksFreeDoM Ours Condition UG MPGD 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The societal impacts are discussed in the conclusions.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: All checkpoints referenced are documented in existing literature, which did not specify any safeguards. Consequently, it is challenging to implement safeguards for these checkpoints.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: The training-based gradient guidance paradigm, such as classifier guidance, is a predominant approach for diffusion guidance. The core objective is to train a time-dependent network that approximates p t (y|x t ) in the RHS of (2), and to utilize the resulting gradient as guidance. The most well-known example is classifier guidance, which involves training a classifier on noisy images. However, classifier guidance is limited to class conditions and is not adaptable to other forms of control, such as image and text guidance. To address this limitation, there are two main paradigms. The first involves training a time-dependent network that aligns features extracted from both clean and noisy images, as described by [24] . The training process is outlined as follows:min ψ E p(x0,xt) d(f ψ (x t , t), f ϕ (x 0 )),where d(•, •) represents a loss function, such as cross-entropy or the ℓ 2 norm. If time-dependent networks for clean images are already available, training can proceed in a self-supervised fashion without the need for labeled data. The second paradigm, as outlined by [21] , involves training an energy-based model to approximate p t (y|x t ). The training process is described as follows:min ψ E p(x0,xt) |ℓ(f ψ (x t , t), y) -ℓ(f ϕ (x 0 ), y)|.However, it is observed in [27] that none of these methods can accurately approximate the true energy in (4). The authors of [27] propose an algorithm to learn the true energy. The loss function is a contrastive lossmin ψ E p(x i 0 ,x i t ) exp(-ℓ(f ϕ (x 0 ), y)) - K i=1 log exp(ℓ(f ψ (x i t , t), y i ) K j=1 exp(-ℓ(f ψ (x i t , t), y i )), where (x i 0 , x i t ) are K paired data samples from p(x i 0 , y i ). Theorem 3.2 in [27] proves that the optimal f ψ * satisfied that ∇ xt ℓ(f ψ * (x i t , t), y i ) = ∇ xt p t (y|x t ). Although this paper focuses on training-free guidance, the findings in this paper can be naturally extended to all training-based gradient guidance schemes. Firstly, the issue of adversarial gradients cannot be resolved without additional training; hence, all the aforementioned methods are subject to adversarial gradients. Empirical evidence for this is presented in Fig. 2 , which illustrates that the gradients from an adversarially robust classifier are markedly more vivid than those from timedependent classifiers. Consequently, it is anticipated that incorporating additional adversarial training into these methods would enhance the quality of the generated samples. Secondly, since these methods are dependent on gradients, employing a more sophisticated gradient solver could further improve their NFEs.Adversarial attacks and robustness constitute a fundamental topic in deep learning [4] . An adversarial attack introduces minimal, yet strategically calculated, changes to the original data that are often imperceptible to humans, leading models to make incorrect predictions. The most common attacks are gradient-based, for example, the Fast Gradient Sign Method (FGSM) [13] , Projected Gradient Descent (PGD) [29] , Smoothed Gradient Attacks [1] , and Momentum-Based Attacks [9] . An attack is akin to classifier guidance or training-free guidance, which uses the gradient of a pre-trained network for guidance. Should the gradient be adversarial, the guidance will be compromised. This paper establishes the relationship between training-free loss-guided diffusion models and adversarial attacks in two ways. Firstly, we prove that training-free guidance is more sensitive to an adversarial gradient. Secondly, in Section 4.2, we demonstrate that borrowing an adaptive gradient scheduler can improve convergence. The optimizers from adversarial attack literature may also expedite the convergence of the diffusion ODE.We use the following training-free diffusion guidance methods as baselines for comparison:• Universal Guidance (UG) [2] employs guidance as delineated in ( 5) and uses time-travel strategies outlined in Algorithm 3 to enhance performance. The time-travel trick is used for all time steps t. UG also utilizes backward guidance, which takes multiple gradient steps at each time step. • FreeDoM [49] is also founded on ( 5) and time-travel trick. In addition, FreeDoM incorporates a time-dependent step size for each gradient guidance and judiciously selects the diffusion step for executing time-travel trick. • Loss-guided Diffusion with Monte Corlo (LGD-MC) [37] utilizes guidance from ( 6) and we set n = 10 in the experiments. • Manifold Guided Preserving Diffusion (MPGD) [16] takes the derivative with respect to estimated clean imageE[x 0 |x t ] instead of x t . Let x 0|t = E[x 0 |x t ],MPGD steps are expressed as following:∇ xt log p t (y|x t ) := - √ α t-1 ∇ x 0|t log exp(-ℓ(f ϕ (x 0|t ), y) .MPGD-Z adopts an additional auto-encoder to preserve manifold constraints. The details procedures of MPGD-Z are described in Algorithm 3 of [16] .For the process of motion diffusion, the application of both MPGD-Z and MPGD-AE is precluded due to the absence of pretrained auto-encoders specific to motion diffusion. An implementation of MPGD without projection (MPGD w/o proj) was attempted for motion diffusion; however, it was unsuccessful in accurately navigating towards the target. This failure is attributed to the presence of spurious correlations within the targeting loss specific to MPGD, a phenomenon not observed in the other baseline methodologies. Sepcifically, the gradient formulation in MPGD is detailed as follows:grad_MPGD = 2(y target -x 0|t ) if t == T 0 otherwiseThe gradient in FreeDoM and other methods is given bygrad_DPS = 2(y target -x 0|t ) • I + σ 2 t ∇ 2 log p t (x t ) √ α t .Analysis of the aforementioned equations reveals that, within the MPGD framework, only the final motion step is influenced by the gradient, a characteristic not shared by alternative methodologies.Consequently, this exclusive focus on the last step results in disproportionately strong guidance at this juncture, while earlier steps suffer from a lack of directional input. This imbalance may adversely affect the overall quality of the samples produced. Empirical observations substantiate that MPGD struggles to achieve targeted outcomes when a nuanced adjustment of step size is required. Given these limitations, MPGD has been excluded from the comparative analysis presented in Table 3 .We follow the prompts and evaluation settings in [37] . The prompts are (i) "the person is walking backwards"; (ii) "a person walking around in a balance beam"; (iii) "the person is walking"; (iv) "the person is jogging". We consider three different directions for each prompt, and each direction has 10 random seeds, the metrics are then averaged together over the 30 synthesized motions.It has been demonstrated in [7] that, given a fixed x 0 , the distribution of the noisy data x t is concentrated on a spherical shell. An extension of this theorem presented in the "High Dimensional Probability" textbook by Vershynin [43] elucidates that the conditional distribution q(x 0 |x t ) also exhibits concentration on a spherical shell, provided that its coordinates are sub-Gaussian.Theorem C.1. (Theorem 3.1.1 in [43] ) Denotex 0 = x t + g ∈ R n , where g = (g 1 , • • • , g n ). Assume that g i are independent identically distributed, E[g 2 i ] = r 2 tn and there exists constant c 1 such that P[g i ≥ t] ≤ exp(-c 1 t 2 ), then we haveP[|∥x t -x 0 ∥ 2 2 -r 2 t | ≥ t] ≤ exp(-c 2 nt 2 ), where c 2 is a constant.Theorem C.1 establishes that the distribution q(x 0 |x t ) exhibits concentration on a spherical shell at an exponential rate. For high-dimensional data, such as images, it is reasonable to infer that x 0 is predominantly situated on this spherical shell.Algorithm 3 Time Travel1: for t = T, • • • , 1 do 2: for i = 1, • • • , s do 3: x i t-1 = DDIM with Guidance(x i-1 t ) 4:if i < s then 5:β t = α t /α t-1 , n ∼ N (0, I)6:x i t = √ β t x i t-1 + √ 1 -β t n 7:end if 8:x 0 t-1 = x s t-1 9:end for 10: end for The technique of time-travel, also referred to as "resampling", has been proposed as a solution to complex generative problems [28] , facilitating successful training-free guidance in tasks such as CLIP-guided ImageNet generation and layout guidance for stable diffusion, as illustrated in Figure 2 of [49] and Figure 8 of [2] , respectively. The procedure of the time-travel trick is shown in Algorithm 3, which involves recursive execution of individual sampling steps.Two-stage convergence: We plot the convergence curve of more images in Figure 5 . The diffusion model is ImageNet diffusion and the guidance network is ResNet-50. The convergence is two-stage: the objective value oscillates when t is large, followed by a swift decrease, which verifies Proposition 3.1. Misaligned gradients: For quantitative experiments, we use robust ResNet-50 to assess the presence of misaligned gradient. Since ResNet-50 serves as our guidance network, a large loss gap between the standard ResNet-50 and the robust variant indicates a severe adversarial gradient issue. In Table 4 , the values represent the loss value. The columns denote the guidance network type, while the rows indicate the loss tested on either ResNet-50 or Robust ResNet-50. The values reported are the average over 1000 images, providing a quantitative evaluation. When using ResNet-50 for guidance, its loss on ResNet-50 is as low as that of real images from the same class (value 5.91). However, the loss for Robust ResNet-50 is significantly higher (value 6.17), suggesting susceptibility to misaligned gradients. By employing ResNet-50 with random augmentation for guidance, we observe a marked reduction in this gap (from 5.91 to 5.98), underscoring the effectiveness of our proposed method.Slower Convergence: Table 5 aims to compare the convergence of training-free guidance with training-based guidance, specifically PPAP [12] . Given that classifier guidance presents a simple scenario and all methods converge relatively fast, it tends to obscure the observations. Consequently, we have directed our attention towards the more complex task of segmentation map guidance in Table 1 . The columns in Table 5 list different methods, while the rows indicate the sampling steps. The values represent the "distance" mentioned in Table 1 and are averaged over 1000 images under the same conditions as those in Table 1 . Our observations reveal a significant discrepancy in the objective values between the training-free FreeDoM and the training-based PPAP [12] , particularly at a lower number of steps (20 or 50 steps), which indicates slower convergence. However, when we incorporate the Polyak step size into FreeDoM, this convergence issue is substantially mitigated. 1 .Given the multiple invocations of the guidance network necessitated by random augmentation (RA), concerns regarding the efficiency of this approach are understandable. However, it is important to note that, compared to the diffusion backbone, the guidance network exhibits a more lightweight architecture, thereby mitigating any significant increase in computational demand. To empirically illustrate this point, we present the computation times associated with varying degrees of augmentation in Table 6 . In the conducted experiments, we set the cardinality of the set T to 10, thereby having little impact on the inference time. These experiments were conducted on a single NVIDIA A100 GPU. 5) can be written as the following:EQUATIONwhere the last equality follows the variance of Tweedie's formula Cov[x 0 |x t ] = σ 2 t (I + σ 2 t ∇ 2 log p t (x t )) [10] . For the first condition, the Lipschitz constant satisfies|ℓ t (x 1 ) -ℓ t (x 2 )| ≤ L f √ α t ∥x 1 -∇ x1 log p t (x 1 ) -x 2 + ∇ xt log p t (x 2 )∥ ≤ L f (1 + L p ) √ α t ∥x 1 -x 2 ∥,and the PL constant satisfiesℓ t (x t ) ≤ 1 µ ∂ℓ ∂ x0 2 = 1 µ ∥∇ xt ℓ • Cov -1 (x 0 |x t )σ 2 t √ α t ∥ 2 ≤ σ 4 t α t µλ 2 min ∥∇ xt ℓ∥ 2 .The second and third conditions directly follow Lemma D.3. Lemma D.3. (Linear Convergence Under PL condition; [22] ) Denote x 0 as the initial point and x t as the point after t gradient steps. If the function is L-Lipschitz and µ-PL, gradient descent with a step size η = 1 L converges to a global solution with ℓ(x t ) ≤ (1 -µ/L) t ℓ(x 0 ).Proof. The proof of this theorem is based on the proof of Lemma 1 in [34] . By the definition of expectation, we havef (x) = E ϵ∼N (0,I) [f (x + σ t ϵ)] = (f ⊛ N (0, σ 2 t I))(x) = 1 (2πσ 2 t ) n/2 R n f (z) exp - 1 2σ 2 t ∥x -z∥ 2 dz.We then show that for any unit direction u, uT ∇ f (x) ≤ 2 πσ 2 t . The derivative of f is given by ∇ f (x) = 1 (2πσ 2 t ) n/2 R n f (z)∇ exp - 1 2σ 2 t ∥x -z∥ 2 dz = 1 (2πσ 2 t ) n/2 σ 2 t R n f (z)(x -z) exp - 1 2σ 2 t ∥x -z∥ 2 dz.Thus, the Lipschitz constant is computed asu T ∇ f (x) ≤ C (2πσ 2 t ) n/2 R n |u T (x -z)/σ 2 t | exp - 1 2σ 2 t ∥x -z∥ 2 dz = C (2πσ 2 t ) 1/2 ∞ -∞ |s| exp - 1 2 s 2 ds = 2 πσ 2 t .Similarly, for the Lipschitz constant of the gradient, we have∥∇ 2 f (x)∥ op ≤ ∥∇ 2 f (x)∥ 2 ≤ C √ 2πσ t • σ 4 t ∞ -∞ s 2 exp(- 1 2 s 2 /σ 2 t )ds + ∞ -∞ σ 2 t exp(- 1 2 s 2 /σ 2 t )ds = 2C σ t .D.4 Proof for Proposition 3.3Proof. We first analyze the discretization error of a single step from time s to t. We denote the update variable for DDIM is x * t and the optimal solution of the diffusion ODE at time t as x * t . Letσ t = √ 1 -α t , λ t = 1 2 log( αt 1-αt) and h = λ t -λ s . According to (B.4) of [26] , the update of DDIM solver is given byEQUATIONA similar relationship can be obtained for the optimal solutionEQUATIONWe then bound the error between x t and x * t . We have∥x * t -x t ∥ ≤ α t α s ∥x * s -x s ∥ + σ t (e h -1)∥u(x * s , s) -u(x s , s)∥ + O(h 2 ) ≤ α t α s ∥x * s -x s ∥ + σ t (e h -1)L∥x * s -x s ∥ + O(h 2 ) ≤ O((1 + L)∥x * s -x s ∥) + O(h 2 )If we run DDIM for M steps, h max = O(1/M ), and we achieve the discretization error bound for DDIM algorithm:∥x 0 -x * 0 ∥ ≤ O(M (1 + L) M h 2 max ) = x * 0 + O((1 + L M )/M ).Proof. By the definition of expectation, we havef (x) = E ϵ∼p(ϵ) [f (x + σ t ϵ)] = (f ⊛ p)(x) = R n f (z)p(x -z)dz.Then we compute the Lipschitz constantu T ∇ f (x) ≤ C R n ∥∇p(x -z)∥ 2 dz = C R n ∥∇p(z)∥ 2 dz.As for the gradient Lipschitz constant, we haveL = ∥∇ 2 f (x)∥ op ≤ C R n ∇ 2 p(x -z)dz op ≤ C R n ∥∇ 2 p(z)∥ op dz,where ∥ • ∥ op is the operator norm of a matrix. See "TeX Source/videos.pptx".LGD Justification: All experiments were conducted on a single A100 GPU, although less powerful GPUs, such as the V100 with 16GB of memory, are sufficient for replicating the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: We confirm that the research conducted in the paper conform.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The code is available at https://github.com/BIGKnight/Understanding-Training-free-Diffusion-Guidance 


Decision: The reviewers and I are in agreement that the paper presents a well-isolated study and contribution on using augmentations and Polyak step sizes for improving Training-free Loss-based Diffusion Guidance.

Reviews: review: paper_summary: paper_summary: The paper studies training-free loss-based diffusion guidance. in comparison to classifier-based guidance. First, the paper examines several drawbacks of loss-based guidance, including (1) while successful minimization of the loss can be achieved, it does not guarantee successful guidance, (2) loss-based gradients can be misaligned with the desired guidance, in contrast to the time-dependent classifier-based gradients with improved smoothness properties, (3) the superior smoothness of classifier-based guidance also means that loss-based guidance takes longer to converge, incurring more function estimations.

In response to those drawbacks, the authors propose two techniques to improve loss-based guidance: (1) random augmentations to improve its smoothness, and (2) Polyak step size to further improve robustness to misalignment between initialization and the specified condition for generation. Three applications are considered for evaluation, where the proposed technique is implemented on top of FreeDoM.
strength_weakness: strength_weakness: Nothing major, though the presentation of Section 3 leaves something to be desired. I'd need to see it after some polishing before I can be satisfied that I fully understand what's going on. See the comments below.
questions: questions: Technical:
========
- Section 3.2 and elsewhere:
  - Recommend to use terms along the lines of "misaligned gradients" rather than "adversarial gradient."
    - I can see it helps to leverage earlier insights from adversarial robustness, e.g., regarding the role of Lipschitz constants, but I worry that there's no benefit to confusing the behavior in question with adversarial robustness considerations. If anything, those gradients were not chosen to maximize deviations w.r.t. any particular objective, but just happen to be misaligned.
    - This suggestion seems to match the narrative on L166-168.
    - Not sure if *adversarial* on L258 is related to this discussion, but just pointing it out.
  - I'm not sure what's exactly meant by "non-Lipschitz" functions
    - A function that's, say, 200-Lipschitz or only 2-Lipschitz is still not 1-Lipschitz.
    - It seems that Proposition 3.2 only needs the upper bound on $\ell$, but need not make any statements about its smoothness.
    - Similarly, L149 can simply state that adding Guassian noise improves smoothness, which is pretty intuitive.
    - It would help to double-check if Proposition 3.2 follows from known standard results, e.g., randomized smoothing which seems to be what $\hat{\ell}$ is about. See Assumption A in
      - Duchi, John C., Peter L. Bartlett, and Martin J. Wainwright. "Randomized smoothing for stochastic optimization." SIAM Journal on Optimization 22, no. 2 (2012): 674-701.
    - Same goes for Proposition 4.1.
    - is it true that Proposition 3.2 is a special case of Proposition 4.1?
- L253: Is there a way to quantify this mismatch between the condition and the input?
- I can't readily place where time travel factors into the presentation and/or the experiments. Is it only mentioned in passing? Is the related analysis in the appendix only included as a bonus?

Presentation:
==========
Section 1:
  - L43: Please cite Appendix E in [25] specifically, since the paper may otherwise seem to be concerned with a different problem altogether, i.e., offline reinforcement learning.
Section 2:
  - Suggest to flatten the section by using bold headers as used later. (Usually looks better than an absent opening paragraph.)
  - Section 2.1 could use some citations, also to clarify which formulation or model is adopted in this paper, and whether the contributions apply to other models as well.
  - L84: Please fix `fastRCNN` and include a citation.
Section 3:
  - Proposition 3.1
    - Recommend to at least provide an intuitive definition of PL. Otherwise, I'd recommend to present an *(informal)* version of the statement that's optimized for readability, deferring the formal version in its entirety to the appendix.
  - Figure 1: would help to explain that a wrong image was obtained in (b).
  - L132: Seems like a good place to start a new paragraph at "Furthermore"
  - L172: Recommend to cite a textbook.
  - L179: Promposition 3.3
    - Is it necessary to reuse the symbol $g$?
    - $g(x_t, t)$ is L-Lipschitz w.r.t. which parameter?
    - It is not clear how the discretization error is defined here. Some context would help.
    - Is it necessary to use the symbol $h_{\text{max}}$? It seems to have nothing to do with the function $h(x_t, t)$.
    - Is $O(h_{\text{max}} + Lh_{\text{max}})$ the same as $O((L+1)h_{\text{max}})$?
    - After a second reading, it's still not clear to me how the discretization error relates to the rate of convergence.
Section 4:
  - Algorithm 1 & 2: Suggest to highlight Tweedie's formula
  - L189: Please include a citation for the Polyak step size.
  - L191: Is it really the case that the analysis presented "completes the picture" for improving training-free guidance? If so, it would help to describe what it takes to complete the picture, then explain how the analysis answers to that. It would also help to highlight any remaining gaps or extensions for future work.
  - L206: Recommend to give names to those integrations defining the Lipschitz bounds.

Nitpicking:
========
- Recommend to reduce redundancy:
  - L1: Adding additional control -> Adding / Controlling / Guiding
  - L19: the generation / the synthesis, as well as the creation -> generation
  - L32: zero-shot generalization to novel conditions -> zero-shot generalization / generalization to novel conditions
  - L38: employ pre-trained networks, designed for clean images [..] checkpoints for these networks pretrained on clean images are -> employ networks pretrained on clean images [..] pretrained networks are
- Recommend to improve consistency in terminology and notation:
  - pre-trained and pretrained
  - clarifies the mystery and resolves the mystery
- Suggest to favor technical terms:
  - "mystery" of guidance weights -> role of guidance weights, principles for choosing guidance weights, intricate interplay between guidance weights and the guidance function and time.
  - L190-191: "trick" -> approach / algorithm / heuristic
- L99: Despite its intuitive -> Despite being intuitive
- L100: considers -> consider
- L100: with Gaussian dist. -> with a Gaussian dist.
- L102: for one-dimension dist. -> for a one-dimensional dist.
- L130: referred as -> referred to as
- L195: incorporating them into -> passing them to
- L212: emerges -> emerge
- L213: efficiency -> efficacy?
limitations: limitations: Although the authors seem to have aimed to complete the picture for training-free loss-based guidance, covering both its drawbacks and potential improvement strategies, this picture is still not clear to me.

For example, it's not clear what's best possible under reasonable conditions on the loss employed for guidance, e.g., how it compares to the (time-dependent) classifier models that can otherwise be employed. I suspect this has something to do with the limitations of time travel. It would be nice to attempt to show some lower-bounds on the errors incurred through time travel.
paper_summary: paper_summary: This paper examines the mechanisms and limitations of training-free guidance for diffusion models and develops a collection of techniques to overcome the limitations accompanied by both theoretical and empirical results.
strength_weakness: strength_weakness: 1.	The paper misses the recent literature in training-free diffusion model, such as RED-diff [1] and DSG [2].
2.	Although the theoretical analysis in this paper illustrates the problems in training-free guidance, the proposed solutions are not particularly novel. Similar ideas have been applied in previous works such as LGD-MC and DSG.

[1] Mardani M, Song J, Kautz J, et al. A variational perspective on solving inverse problems with diffusion models[J]. arXiv preprint arXiv:2305.04391, 2023.

[2] Yang L, Ding S, Cai Y, et al. Guidance with spherical gaussian constraint for conditional diffusion[J]. arXiv preprint arXiv:2402.03201, 2024.
questions: questions: 1.	How is the performance of MPGD and LGD-MC modified with the random augmentation and Polyak step size?
2.	How is the proposed method compared with DPS, DSG and RED-diff?
limitations: limitations: The authors have clearly presented the limitations in the paper.
paper_summary: paper_summary: This paper explores the theoretical aspects of training-free loss-based guidance mechanisms in diffusion models and improves them from theoretical findings. The first results are in explaining why the guidance strength depended on $\sqrt{\alpha_t}$ worked well in FREEDOM. Then, the authors explore the adversarial gradients, which result in a lower loss but produce generations with the incorrect condition. Specifically, they demonstrated that a time-dependently trained network with Gaussian noises is smoother compared to a general network. This smoothness can cause a slowdown of guidance. From these theoretical results, they propose to utilize random augmentation to increase the smoothness of the off-the-shelf network and improve the convergence speed with Polyak step size, which accelerates convergence by adaptive gradient step size. Experimental results show that their method can outperform loss-based guidance methods including Universal Guidance, Loss-Guided Diffusion, FreeDom, and MPGD-Z.
strength_weakness: strength_weakness: - My first major concern is missed baselines and related works. This work aims to improve off-the-shelf models’ guidance with the training-free based method, and [A] deals with the same configuration. [A] elucidates the design space of off-the-shelf guidance, and smoothed classifier, joint classifier guidance, and guidance schedules are explored in their work. I think that the smoothed classifier concept in [A] is very similar to random augmentation for the smoother network in this work, and similar guidance schedule schemes are explored in [A] and this work. Therefore, [A] should be discussed in this work and be compared in the experimental section. Also, I found [C] and [D].
- In the other line of work, [B] proposes the utilization of off-the-shelf models by finetuning that model to be the time-dependent network. Although [B] can be categorized into the training-required method, it would be better to discuss this kind of training-required method and compare it in the experimental section. Showing the performance gaps between training-free and training-needed methods is needed to validate the effectiveness of the proposed method.
- [A] and [B] choose the imagenet classifier guidance with imagenet trained diffusion models and various diffusion models use imagenet class conditional generation as the core benchmark. However, this work does not incorporate this major benchmark in their benchmark, so it can make reader hard to recognize their absolute performance.
- There are no details about the Polyak Step size, and if Algorithm 2 is all of the details of the Polyak Step Size, there have been many works to utilize normalized gradients and I felt that this is not novel.
questions: questions: ## Typo
- Line 875 in Appendix: $||f(x_2)-f(x_2)||$ ->  $||f(x_2)-f(x_1)||$
- Equations are referred to as (number). I am very confused to understand whether (number) is an equation or a bullet point. How about using Eq. (number)?


## References

- [A] ELUCIDATING THE DESIGN SPACE OF CLASSIFIER GUIDED DIFFUSION GENERATION, ICLR 2024

- [B] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.

- [C] ADJOINTDPM: ADJOINT SENSITIVITY METHOD FOR GRADIENT BACKPROPAGATION OF DIFFUSION PROBABILISTIC MODELS, ICLR 2024.

- [D] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method, Arxiv 2023.
limitations: limitations: The higher number of NFEs required in training-free guidance and adversarial guidance are mentioned in the limitation section.
paper_summary: paper_summary: This paper explores training-free loss-based diffusion guidance, providing an overview of how training-free guidance functions from an optimization perspective. It analyzes the challenges of training-free guidance, particularly regarding adversarial gradient issues and slower convergence rates. Additionally, the paper proposes two improvements: random augmentation and Polyak step size. These enhancements are validated through experiments across various applications.
strength_weakness: strength_weakness: Although the paper provides a thorough theoretical analysis of why training-free guidance works and doesn’t work, the empirical evidence is insufficient to fully support the claims. The propositions are based on the assumption of Lipschitz continuity, which may not hold for complex deep neural networks, therefore, a more comprehensive quantitative analysis is necessary. Here are my detailed comments:

- The experiment presented in Figure 1 is somewhat confusing. The authors aim to convey three points: (1) the loss curve will oscillate initially and decrease eventually; (2) the guidance weight needs careful selection; (3) adversarial gradients may cause incorrect generated images despite low loss. The first point requires more extensive experiments rather than just two cases. The second point is not clearly inferred from the figure.
- The two limitations of training-free guidance, adversarial gradient, and slower convergence rates, lack quantitative experimental evidence.

Regarding the proposed method, while benchmarking many baselines on various tasks is appreciated, several issues remain unaddressed:

- The difference between UG and FreeDoM in this paper appears to be the selection of diffusion steps for the time-travel trick. However, UG also uses a “backward universal guidance” technique as mentioned in section 3.2 of the original paper [1].
- The authors did not choose classifier-based tasks, which is curious given that the paper analyzes the inferiority of training-free guidance compared to training-based guidance through this kind of task in Sec 3. Comparative experiments showing the gap between these approaches are essential. Including classifier-guided generation tasks would be beneficial to assess the significance of the two tricks’ improvements, and training-based baselines can be included in this task.
- The selection of hyper-parameters is concerning. With numerous hyper-parameters involved, how are the optimal values (e.g., step size, LGD-MC Monte Carlo samples, diffusion step range for time-travel) chosen? How is it ensured that baselines are implemented optimally?
- The random augmentation trick may have limitations:
  - While image augmentation techniques are mature, effective augmentation tricks for other domains (e.g., text, AI4Science, offline RL) may not be available, limiting the trick’s application.
  - Despite the claim that random augmentation does not introduce significant computational overhead, this may not hold true when using a large guidance network, as would be the case with a large, generalist foundation model for scoring samples.



[1] UNIVERSAL GUIDANCE FOR DIFFUSION MODELS. https://openreview.net/pdf?id=pzpWBbnwiJ
questions: questions: - What is the rationale behind choosing the Polyak step size? It appears to be a post-hoc decision without strong intuition.
- Since random augmentation is omitted in MDM, is the only difference between “Ours” and “FreeDoM” the Polyak step size? The loss values in Table 3 show a significant difference between FreeDoM and Ours, and clarification on this point is needed.
- What's the value of $\eta$ set for DDIM?
limitations: limitations: Yes.

Comments: title: Follow up
comment: Thanks for the timely response. Considering of the theoretical contribution of this paper which helps to understand training-free loss-based diffusion guidance better, I raise the score to a 5.
We greatly appreciate your insightful suggestions and comments, which have significantly contributed to the refinement of our paper. We are also thankful for your acknowledgment of our rebuttal efforts.

**Comment 1:** Regarding Table 3, it is designed to evaluate the adversarial gradient issue in training-free guidance. To assess the presence of adversarial gradients, we utilized an adversarially robust ResNet-50 [33], which is trained to resist adversarial gradients. Since ResNet-50 serves as our guidance network, a large loss gap between the standard ResNet-50 and the robust variant indicates a severe adversarial gradient issue. In Table 3, the values represent the loss value. The columns denote the guidance network type, while the rows indicate the loss tested on either ResNet-50 or Robust ResNet-50. The values reported are the average over 1000 images, providing a quantitative evaluation. When using ResNet-50 for guidance, its loss on ResNet-50 is as low as that of real images from the same class (value 5.91). However, the loss for Robust ResNet-50 is significantly higher (value 6.17), suggesting susceptibility to adversarial gradients. By employing ResNet-50 with random augmentation for guidance, we observe a marked reduction in this gap (from 5.91 to 5.98), underscoring the effectiveness of our proposed method.

Table 4 aims to compare the convergence of training-free guidance with training-based guidance, specifically PPAP [D]. Given that classifier guidance presents a simple scenario and all methods converge relatively fast, it tends to obscure the observations. Consequently, we have directed our attention towards the more complex task of segmentation map guidance in Table 1. The columns in Table 4 list different methods, while the rows indicate the sampling steps. The values represent the "distance" mentioned in Table 1 and are averaged over 1000 images under the same conditions as those in Table 1. Our observations reveal a significant discrepancy in the objective values between the training-free FreeDoM and the training-based PPAP [D], particularly at a lower number of steps (20 or 50 steps), which indicates slower convergence. However, when we incorporate the Polyak step size into FreeDoM, this convergence issue is substantially mitigated.

**Comment 2:** As for the second concern, we are thankful for your acknowledgment of our efforts in the rebuttal. We have also conducted comparisons with training-based methods, as illustrated by PPAP in Table 1 of the PDF provided in the global rebuttal. PPAP extends classifier guidance to accommodate a broader range of conditions by fine-tuning the pretrained model on clean images to create different experts, each responsible for a specific range of noise level. The gradients from the corresponding experts are then used for guidance at corresponding noise levels. To optimize performance of PPAP, we have set the number of experts to $10$, which is the maximum reported in [D]. As indicated in Table 1, our proposed method shows performance on par with the training-based PPAP. With respect to the reviewer's request for additional experiments under class conditions, we are currently conducting these experiments. However, due to the high number of required samples and the author-reviewer discussion coming to an end, it is possible that they cannot be completed before the reviewer-author discussion period closes. We assure you that the results of these experiments will be included in the revised manuscript.

**Comment 3:** Regarding the third question, the effectiveness of random augmentation is supported by Proposition 4.1. This proposition does not rely on the assumption of Lipschitz continuity or any specific distribution for the augmentation, suggesting its broad applicability across various modalities. Consequently, the theoretical foundation guarantees the effectiveness of random augmentation in AI4Science or RL without imposing any strong assumptions. As we approach the end of the author-reviewer discussion and considering AI4Science or RL represents a novel application setting for our paper, it is regrettably unlikely that additional experiments can be completed prior to the submission deadline. Nonetheless, we are committed to including these experiments in the revised manuscript and we appreciate the reviewer's understanding in this matter.
Thank you to the authors for the detailed responses. I have some follow-up comments:

1. I apologize for not fully understanding the meaning of Table 3 and Table 4 in the uploaded PDF. Could you please clarify what the values in these tables represent? I would appreciate a more detailed explanation of the settings for each table.

2. Regarding the classifier-based tasks, I appreciate that the authors chose to evaluate their method on tasks where FreeDoM and LGD-MC have shown strong performance. However, I still believe that using tasks involving classifier guidance would be more appropriate. This would allow for more direct comparisons with classifier-based methods (or even classifier-free guidance), making both quantitative comparisons and qualitative analyses easier.

3. Thank you for highlighting the presence of augmentation methods in molecule and RL domains. My concern is that it's difficult to justify the effectiveness of all augmentation methods in training-free, loss-based diffusion guidance, as the efficacy of augmentation can vary significantly across different domains.
We are immensely grateful for the time and attention you have invested in reviewing our manuscript. Your detailed comments and constructive feedback have not only highlighted areas that needed improvement but have also deepened our understanding. We want to assure you that we take your comments seriously and are currently in the process of revising our manuscript accordingly.
Thanks for responding to my comments.

I recommend to move Proposition 4.1 to the preliminaries section under a new subsection, e.g. can be titled Randomized Smoothing, with adequate citations. Proposition 3.2 should follow immediately by taking $p$ to be $\mathcal{N}(0, I)$, and can be stated as a corollary.
Thank you for your comprehensive feedback and insightful review. Your observations have been extremely beneficial in refining our manuscript. We are deeply appreciative of your recognition of our theoretical contributions and our responses to the critiques. As further detailed in the second point of our response, the methodologies we introduce are distinct from LGD-MC and DSG in their motivations and methodology, presenting complementing strategies. The experiments (Table 1 of the PDF in the global response) further indicate that the proposed methods can augment the performance of these existing works. Once again, we express our sincere thanks for your valuable comments and prompt reply.
Thank you for your thorough review and insightful suggestions. Your comments have been invaluable in refining our paper. We appreciate your positive reception of our rebuttal and are glad that our response has addressed your concerns. We are committed to carefully revising the manuscript according to your feedback.
Thanks for your responses and the efforts on the additioanl experiments. Although the theoretical contribution of this work should be acknowledged, the novelty of this paper is still limited in the reviewer's opinion. In that case, I am sticking to my score.
We apologize for the oversight in not including references for Weakness 1 in our previous submission. The references for the rebuttal of Weakness 1 are as follows:

[A] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

[B] Accelerating convergence of score-based diffusion models, provably
My concerns are fully addressed. Accordingly, I raised my rating as 5