Understanding and Improving Training-free Loss-based Diffusion Guidance
Authors: Yifei Shen
XINYANG JIANG
Yifan Yang
Yezhen Wang
Dongqi Han
Dongsheng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Training-free guidance
universal guidance
motion diffusion
Primary area: diffusion_based_models
TL;DR: This paper examines the mechanisms and limitations of training-free guidance for diffusion models, proposing methods to address these challenges effectively.

Abstract: Adding additional guidance to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free loss-based guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of training-free guidance, as well as overcome its limitations. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free guidance is more susceptible to misaligned gradients and exhibits slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.

Understanding and Improving Training-free Loss-based Diffusion Guidance
Yifei Shen' Xinyang Jiang' Yifan Yang',
Yezhen Wang² Dongqi Han' Dongsheng Li'
Microsoft Research Asia
National University of Singapore
Abstract
Adding additional guidance to pretrained diffusion models has become an in- creasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-fit free bass-based guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer free lunch in diffusion guidance. this paper, aim develop deeper understanding of training-free guidance, as well as overcome its limitations. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theo- retically demonstrate that training-free guidance is more susceptible to misaligned gradients and exhibits slower convergence rates compared to classifier guidance. We then introduce collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy these techniques.
Introduction
Diffusion models represent class of powerful deep generative models that have recently broken the long-standing dominance of generative adversarial networks (GANs) (8). These models have demonstrated remarkable success variety of domains, including the generation of images and
videos in computer vision [33 (3, the synthesis of molecules and proteins in computational biology well as the creation trajectories and actions in the field of reinforcement learning (RL)
20 52,
21
One critical area of research in the field of diffusion models involves enhancing controllability, such as pose manipulation in image diffusion (50), modulation of quantum properties in molecule diffusion (20] and direction of goal-oriented actions in RL diffusion (23] The predominant techniques
for exerting control over diffusion models include classifier guidance and classifier-free guidance.
Classifier guidance involves training time-dependent classifier to map noisy image, denoted as It, to specific condition y. and then employing the classifier's gradient to influence each step of the diffusion process [8]. Conversely, classifier-free guidance bypasses the need for classifier by training an additional diffusion model conditioned on (18). However, both approaches necessitate extra training integrate the conditions. Moreover, their efficacy is often constrained when the data-condition pairs are limited and typically lack the zero-shot generalization capability.
Recently, several studies 2 49 37 have introduced training-fre guidance that builds upon the concept of classifier guidance. These models eschew the need for training a classifier on noisy images; instead, they estimate the clean image from its noisy counterpart using Tweedie's formula and then employ pretrained networks, designed for clean images, to guide the diffusion process. Given that
"Contact yiweeisseeeeeiicoooottteee com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

checkpoints for these networks pretrained on clean images are widely accessible online, this form of guidance can be executed zero-shot manner. unique advantage of training -reee guidance is that it can be applied to universal control formats, such as style, layout, and FacelD 2
49 37 without any
additional training efforts. Furthermore, these algorithms have been successfully applied to offline reinforcement learning. enabling agents to achieve novel goals not previously encountered during training. In contrast to classifier guidance and classifier- free guidance, is proved in Appendix of [27 that training-free guidance does not offer an approximation to the exact conditional energy. Therefore, from a theoretical perspective, it intriguing to understand how and when these methods succeed or fail. From empirical standpoint, it is crucial to develop algorithms that can address and overcome these limitations.
This paper seeks to deepen the understanding of training-free free guidance by examining its mechanisms and inherent limitations, as well overcoming these limitations. Specifically, our major contributions can be summarized follows:
How does training-free guidance work? Although exact conditional energy is difficult to approximate in training- firee manner, from the optimization standpoint, we show that training-fit free guidance can effectively decrease the guidance loss function. The optimization perspective clarifies the mystery of why the guidance weights should be meticulously designed relation to the guidance function and diffusion time, as observed [49].
When does training-free guidance not work? We theoretically identify the susceptibility of training-freee guidance to misaligned gradient issues and slower convergence rates. We attribute these challenges decrease in the smoothness of the guidance network in contrast to the classifier guidance.
Improving training-free guidance: We introduce random augmentation to alleviate the misaligned gradient and Polyak step size scheduling to improve convergence. The efficacy these methods is empirically confirmed across various diffusion models (i.e., image diffusion and motion diffusion) and under multiple conditions (i.e., segmentation, sketch, text, object avoidance, and targeting
2 Preliminaries
2.1 Diffusion Models
Diffusion models are characterized by forward and reverse processes. The forward process, occurring over a time interval from O to T, incrementally transforms an image into Gaussian noise. On the contrary, the reverse process, from T back to O, reconstructs the image from the noise. Let ar represent the state of the data point at time I; the forward process systematically introduces noise to the data by following a predefined noise schedule given by æ, = Vax,a0 + orEt, where de E [0. 1] monotonically decreasing with t, o, = VI- at, and E, ~ NO, I) is random noise. Diffusion models use neural network to learn the noise at each step:
min ee. lE (x E min E...,, t) + V log Pı (xx)
where pr(x)) is the distribution of æ,, The reverse process is obtained by the following ODE:
dæ: [(1)=,
(²)
g²(1)
V n, lo..)( f(1),, + En c !),
(1)
dt
2
20:
where ft) log voe (t) dof
d va di de
of. The reverse process enables generation as
dt converts a Gaussian noise into the image.
2.2 Diffusion Guidance
In diffusion control, we aim to sample æn given condition y. The conditional score function is expressed as follows:
log Pa (xaly) = Va, (x + Va, log po
(2)
"The code is available at
https:/ / //gg://///ttbb/ /gttttt......gggggggggggg....eet www/ htmm////KKiiiihhh/nnnnrrsaaannnn -rraoiinn---aaiinn--eeeeee hww.aigg-rrre-..ffuuu-------e---

---

The conditions are specified by the output of neural network, and the energy is quantified by the corresponding loss function. If (+) represents the loss function as computed by neural networks, then the distribution of the clean data is expected to follow the following formula 2 49 37
po y) x po(a exp(-((fo y)).
For instance, consider scenario where the condition is the object location. In this case, fo represents a fastRCNN architecture, and l denotes the classification loss and bounding box loss. By following the computations outlined in [27] we can derive the exact formula for the second term in the RHS of as:
ar log Pe (yylar)) V ve log ppp (mx)) (elp))///x/p//ee (exp((((fo((x))))) y)].
Classifier guidance 8 involves initially training a time-dependent classifier to predict the output of the clean image 3o based noisy intermediate representations æ, during the diffusion process, i.e., to train a time-dependent classifier (ax, such that Su (axe, 1) z La (mo) Then the gradient of the time-dependent classifier is used for guidance, given by V a, log Pe are) == V [f (Ir,, (), y)- This term equals (4) if the loss is cross-entropy.
Training- freee loss- guidance 2 49 6 puts the expectation in inside the loss function:
VR, (og V log (Bpp-( 2)]
(a)
V V
fo az
y
Voz
(5)
where (a) uses Tweedie's 's formula (a (az)) 2 (m)
Leveraging this formula permits the use of pretrained off-the-shelf network designed for processing clean data. The gradient of the last term in the energy function is obtained via backpropagation through both the guidance network and the diffusion backbone.
va:
Analysis of Training-Fri Guidance
3.1 How does Training-free Guidance Work?
On the difficulty of approximating Va, log Pe xx)) in high-dimensional space. Despite being intuitive, (27 has shown that training-free -free guidance in (5) does not offer an approximation to the true energy in The authors of (37] consider to directly approximate (4) with a Gaussian distribution:
V m: log Ex( .) (eeppl/eeppppeppp (exp(=((fo(())) 4)
V log Ee( (eepe)/eepppepepp (ep((=((f(((x)) y)]
(6)
m log
exp(-e(fo(ar)) 9)),
m ((
where (z 2 is chosen (N (Io),, rT and ri is tunable parameter. As demonstrated in [37], the approximation is effective for one-dimensional distribution. However, we find that the approximation denoted by (a) does not extend to high- -igh-nminnii data (e.g., images) if the surrogate distribution is sub-Gaussian. This is due to the well-known high-dimensional probability 4K] that if q has sub-Gaussian coordinates (e.g., iid and bounded), then g( (a:)) tends to concentrate on a spherical shell centered at (a.l))))) (at)) with radius re (details are in Appendix C.I Since the spherical shell represents low-dimensional manifold with zero measure in the high-dimensional space, there is significant likelihood that the supports of p(æp xx) and g( (x;) do not overlap, rendering the approximation (a) ineffective.
phenomenon
Understanding training-free guidance from an optimization perspective. We instead analyze the training-free free guidance from the optimization perspective. Intuitively, in each step, the gradient is taken and the loss of the guidance network decreases. At the initial stage of the diffusion (t is large), the diffusion trajectory can exhibit substantial deviations between adjacent steps and may increase the objective value. So the objective value will oscillate at the beginning. When is smaller, the change the sample is more fine-grae fine-grained, leading a bounded change in the objective value. Therefore, the objective value is guaranteed to decrease when t small, as showing in the next proposition.
Proposition 3.1. Assume that the guidance loss function {(J((rr)) y) is P-PL (defined Defin- tion D.2 in appendix) and Lj Lipsschhtz with respect clean images No. and the score function
3

---

V Pl is Lp- Lipschitz (defined in Defintion D.1 in appendix) with respect to noisy image Dt- Denote Amin as the minimum eigenvalue the (semi)-definite matrix Covlæz | Then the following 1e axtooo log pe(we)
conditions hold: (1) Consider the loss function (e( =
y and denote
Var
1x 11 T.((+l))vaio After one gradient step it 3c 2,
T.p( we have
(a (1- (()))) (2) Consider diffusson process that adheres to bounded change in the Ee(tt)
objective function such that for any diffussion step, i.e., ( S
for some fig < K1, then
k2)
the objective function converges at linear rate, i.e. l-1 ( S (
Ganerated mage Lons Correct Image Loss
5.6
Gene reted Imsge Less
Connec Imags Loss
238 Difiusion
800 Diffuarin Tima
(a) A successful guidance.
(b) A failure guidance.
Figure The classifier loss a successful and a failure guidance example. The target class is "indigo bird".
The proof is given in Ap The Lipschitz continuity and PL conditions are basic assumptions in optimization, and it has been shown that neural networks can locally satisfy these conditions (5). These assumptions, while essential, simplify the proof and results for clarity, similar the assumptions taken in [26). The optimization perspective clarifies the mystery of why the guidance weights (i.e., the step size in optimization) should be carefully selected with respect to the guidance function and time t. For example, in [49] most guidance weights 17 are chosen to be proportional to
yai and dependent on guidance network, which differs from the weights used in classifier guidance and aligns with our step size in Proposition 3.1
Then empirically verify Proposition 3.1 Via experiments in Figure We use ResNet-50 trained on clean images to guide ImageNet pretrained diffusion models. The loss value at each diffusion step is plotted. As reference, we choose 100 images from the class "indigo bird" in ImageNet training set and compute the loss value, which is referred to as "Correct Image Loss" in the figure. The objective value oscillates when is large, followed by a swift decrease, which verifies our analysis. More convergence figures are given in Figure 7 in Appendix.
An intriguing aspect of the theory is that the loss remains low regardless of the success of the guidance, akin to the loss associated with correct images. Figure demonsttrtss this phenomenon: despite the absence of indigo bird in the image, the loss is still minimal. This phenomenon can be attributed to the effect of misaligned gradients, which is explored in detail the following subsection.
3.2 Limitations of Training-free Guidance
In this subsection, examine the disadvantages of employing training-fre free guidance networks as opposed to training-based classifier guidance.
Training-free guidance is more sensitive to the misaligned gradient. Adversarial gradient is significant challenge for neural networks, which refers to minimal perturbations deliberately applicd to inputs that induce disproportionate alterations in the model's output (38). The resilience of model to ad adversarial gradients often analyzed through the lens of its Lipschitz constant [34]. If the model has a lower Lipschitz constant, then the output is less sensitive to the input perturbations and thus is more robust.
In the classifier or training- free guidance, the gradient of the guidance network is added to the image. contrast to yielding a direction that meaningfully minimizes the loss, the adversarial gradient primarily serves to minimize the loss in manner that is not necessarily aligned with the intended guidance direction. As result, we refer the adversarial gradient of guidance network as the misaligned gradient in diffusion guidance.

---

Compared with the off she-lle guidance network used in training- free guidance, time-dependent classifiers are trained on noise-augmented images. Our finding is that adding Gaussian noise improves the Lipschitzness of the guidance network. This transition mitigates the misaligned gradient challenge by inherently enhancing the model's robustness such perturbations, as shown in the next proposition.
Proposition 3.2. (Time- (Time-dependent network more robust and smooth) Given bounded loss function (x)= C, the loss ((x) = BewNio,n)(((r [ote]] is Cy नoद -ipschitzz and V is 2C Lipscchitz
(a) Adversarially robust (b) Time-dependent classi- (c) Off-th ResNet- (d) ResNet-50 with ran- classifier.
fier.
50 classifier.
dom augmentation.
Figure 2: Gradients of different classifiers on random backgrounds. The images in the first row correspond to the target class "cock", and the second row to "goldfinch".
The proof is given in Appendix D33 We then support Proposition 3.2 with both qualitative and quantitative experiments. For qualitative experiments, we present visualizations of the accumulated gradients for both the time-dependent and off-the-shelf time-independent classifiers corresponding to different classes in Figure 2b and Figure 2cl respectively. These visualizations are generated by initializing image with random background and computing 1000 gradient steps for each classifier. For the time-dependent classifier, the input time for the (-th gradient step is 1000 images generated purely by the classifier gradients without diffusion involved. For comparative analysis, we include the accumulated gradient an adversarially robust classifier [35], as shown in Figure 2a which has been specifically trained resist misaligned (adversarial) gradients. The resulting plots reveal a stark contrast: the gradient of the time-dependent classifier visually resembles the target image, whereas the gradient of the time-independent classifier does not exhibit such recognizability. This observation suggests that off-the-shelf time-independent classifiers are prone generating misaligned gradients for guidance compared the time-dependent classifier used in classifier guidance. The quantitative experiments are given in Table in Appendix.
The
Figure provides more intuitive visual explanation of diffusion guidance compared the existing formula- based approaches as shown in 7 The gradient produced by the guidance network represents valid image. When these gradients are incorporated into the images, the diffusion model is able to identify the object and enhance into a clearer and more vivid representation.
Training-free free guidance slows down the convergence of reverse ODE. The efficiency of an algo- rithm in solving reverse ordinary differential equations is often gauged by the number of non-linear function estimations (NFEs) required to achieve convergence. This metric is vital for algorithmic design, as directly relates to computational cost and time efficiency [36]. light this, we explore the convergence rates associated with various guidance paradigms, beginning analysis with reverse ODE framework that incorporates generic gradient guidance term. The formula is expressed
dæ; = f(t + g2(1) (co(E,))) va,,(z,,
dt where -). can be either a time-dependent classifier or a sime-independent classifier with Tweedie's formula. The subsequent proposition elucidates the relationship between the discretization error and the smoothness of the guidance function. Proposition
3.3. Let u(
EB (
Va,v(x,, in hmax
max; / [og[( - log(
Assume we run DDIM solver for M steps and M = 011/hmmll Then the error is bounded by 0((1 L((1+MM)/M) 1/M).
5

---

The proof is given in Appendix D.4] Proposition 3.2 establishes that time dependent classifiers exhibit superior gradient Lipschitz constants compared their off-the-shelf -ineinddeendent counterparts. This disparity in smoothness slows down the convergence for training-free guidance methods, necessi- tating a greater number of NFEs achieve the desired level of accuracy when compared to classifier guidance. To provide quantitative support, we compare the convergence speed of training-hase baane PPAP [12] and training-free FreeDoM Table in Appendix.
4 Improving Training-free Guidance
In this section, we propose to adopt random augmentation to mitigate the misaligned gradient issue, and Polyak step size [15] to mitigate the convergence issue. In addition to these two techniques, method and baselines will also incorporate trick named time travel, often referred to as "restart sampling". and its theoretical framework is detailed in [46].
Algorithm Random Augmentation
Algorithm 2 Polyak Step Size
for 1 T, .0 do
for T, ..... do
- (D+
- (DIM((.))
in 2=-0)co(0,.!) V(t
Tweedie's formula
i 2)-0r60((Fe.t
D. Tweedie's formula
you
9: T ETET
Va, ((o(((()))) T;-1 x1-1 9t
9: Vx, ((fo(())))))))
y)
(mr
æ:-1 = 2t-1
19
end for
end for
4.1 Random Augmentation
As established by Proposition 3.2 the introduction of Gaussian perturbations enhances the Lipschitz property of neural network. A direct application of this principle involves creating multiple noisy instances of an estimated clean image and passing them into the guidance network, a method analogous to the one described in (6). However, given the high-dimensional nature image data, achieving satisfactory approximation of the expected value necessitates impractically large number of noisy copies. To circumvent this issue, we propose an alternative strategy that employs a diverse set of data augmentations in place of solely adding Gaussian noise. This approach effectively introduces perturbations within a lower-dimensional latent space, thus requiring fewer samples. The suite of data augmentations utilized, denoted by T, is derived from the differentiable data augmentation techniques outlined in [51). which encompasses transformations such as translation, resizing, color adjustments, and cutout operations. The details are shown in Algorithm D and the rationale is shown in the following proposition.
Proposition 4.1. (Random augmentation improves smoothness) Given a bounded non- non-Lipschitz loss function (xx), the loss f(x) = [e-p(c)(((x + +)] is C. Jg. [Vp(t)) Lipschitt and its gradient is C' fgm
The proof is shown in Appendix D.5 Echoing the experimental methodology delineated Section we present an analysis of the accumulated gradient effects when applying random augmentation to a ResNet-50 model. Specifically, we utilize a set of |T] = 10 diverse transformations as our augmentation strategy. The results of this experiment are visualized in Figure 2d where the target object's color and shape emerge in the gradient profile. This observation suggests that the implemen- tation of random augmentation can alleviate the misaligned gradient issue. The quantitative effect of random augmentation is given in Table in Appendix. The computational efficiency of random augmentation is further discussed in Appendix C.4
3.2
4.2 Polyak Step Size
we analyzed training-free guidance from the optimization perspective. To accelerate the convergence, gradient step size should be adaptive to the gradient landscape. We adopt Polyak step size, which has near-optimal convergence rates under various conditions [15]. The algorithm is
In Section 3.1

---

FreeDoM
FreeDoM
FreeDoM iPolyal)
FreeDoM (Polyak)
(a) When the initialization is proximate to the specified (b) When the initialization deviates from the condi- conditions, both step sizes perform satisfactorily.
tions, only the Polyak step size guides effectively.
Figure 3: The effects of step size.
shown in Algorithm 2 and the term [ t)| is used to both estimate the gap to optimal values and balance the magnitude of diffusion term and guidance term.
We implement Polyak step size within the context of a training-fres guidance framework called FreeDoM [49] and benchmark the performance of this implementation using the DDIM sampler with steps. As shown in Figure FreeDoM is unable to effectively guide the generation process when faced with significant discrepancy between the unconditional generation and the specified condition. An illustrative example is the difficulty in guiding the model to generate faces oriented to the left when the unconditionally generated faces predominantly orient to the right, shown in FigureB3b This challenge, which arises due to the insufficiency 50 steps for convergence under the condition, is ameliorated by substituting gradient descent with adaptive step size, thereby illustrating the benefits of employing a better step size in the guidance process. The quantitative experiments are given in Table 5 in Appendíx.
Experiments
In this section, we evaluate the efficacy of proposed techniques across various diffusion models and guidance conditions. We compare our methods with established baselines: Universal Guidance (UG) (2). Loss-Guided Diffusion with Monte Carlo (LGD-MC) B77, Training-Free Sedegyy-Guiedd Diffusion Models (FreeDoM) [49], and Manifold Preserving Guided Diffusion (MPGD) [16]. LGD- MC utilizes ( while UG and FreeDoM are built (5). MPGD utilizes auto-encoder to ensure the manifold constraints. Furthermore, time travel trick (Algorithm 3) is adopted in UG, FreeDoM, and MPGD improve the performance. Please refer Appendix
B for details of baselines. For the The method "Ours" is built on
sampling method, DDIM with 100 steps adopted as in (49 37 FreeDoM, with Polyak step size and random augmentation.
5.1 Guidance to CelebA-HQ Diffuusio
In this subsection, we adopt the experimental setup from [49]. Specifically, we utilize the CelebA -HQ diffusion model [19] generate high-quality facial images. We explore three guidance conditions:
segmentation, sketch, and text. For segmentation guidance, BiSeNet [48] generates the facial segmentation maps, with an f2lloss applied between estimated map of the synthesized image and
the provided map. Sketch guidance involves using the method from [45] to produce facial sketches, where the loss function the loss between the estimated sketch of & and the given sketch. For
text guidance, we employ CLIP (32] as both the image and text encoders, setting the loss to be the & distance between the image and text embeddings.
We randomly select 1000 samples each of segmentation maps. sketches, and text descriptions. The comparative results are presented Table [ Consistent with (49,, the time-travel number for all methods is set to = 1. Figure g displays random selection of the generated images. More image samples are provided in the supplementary materials. We find that the basclines failed to guide if the condition differs from unconditionally generated images significantly, discussed in Section
5.2 Guidance to ImageNet Diffusion
For the unconditional ImageNet diffusion, we employ text guidance in line with the approach used in FreeDoM and UG 2 49). We utilize CLIP-B/16 as the image and text encoder, with cosine similarity
7

---

Segmentation maps
Sketches
Texts
Methods
Distancel
FID
Distancel
FID.
Distancel.
FID
UG 22
2247.2
39.91
52.15
47.20
12.08
44.27
LGD-MC 37
2088.5
38.99
49.46
54.47
11.84
41.74
FreeDoM
49
1657.0
38.65
34.21
52.18
11.17
46.13
MPGD-Z 16
1976.0
39.81
37.23
54.18
10.78
42.45
Ours
1575.7
33.31
30.41
41,26
10.72
41.25
Table 1: The performance comparison of various methods on CelebA-HQ with different types of
zero-shot guidance. The experimental settings adhere Table 1 of 49
Methods
LGD-MC
37
UG 22
FreeDoM [49
MPGD-Z Z [6
Ours
CL.IP Scoref
24.3
25.7
25.9
25.1
27.7
Table 2: The performance comparison of various methods on unconditional ImageNet with zero-shot text guidance. We compare various methods using ImageNet pretrained diffusion models with CLIP-B/16 guidance. For evaluating performance, the CLIP score is computed using CLIP-L/14.
serving as the loss function to measure the congruence between the image and text embeddings. To evaluate performance and mitigate the potential for high-scoring adversarial images, we use
CLIP-L/I4 for computing the CLIP score. In FreeDoM and MPGD-Z, resampling is conducted for time steps ranging from 800 to 300, with the time-travel number fixed at 10, described in [49). Given that UG resamples at every step, we adjust its time-travel number 5 to align the execution time with that of FreeDoM. The textual prompts for our experiments are sourced from (25. The comparison of different methods is depicted in Table The corresponding randomly selected images are illustrated in Figure [5] The table indicates that our method achieves the highest consistency with the provided prompts. As shown in Figure 5 LGD-MC and MPGD tend to overlook elements of the prompts. Both UG and FreeDoM occasionally produce poorly shaped objects, likely influenced by misaligned gradients. Our approach addresses this issue through the implementation of random augmentation Additionally, none of the methods successfully generate images that accurately adhere to positional prompts such as "left to" or "below", This limitation is inherent to CLIP and extends to all text-to-image generative models materials.
More image samples are provided in the supplementary
5.3 Guidance to Human Motion Diffusion
In this subsection, extend our evaluation to human motion generation using the Motion Diffusion Model (MDM) [40]. which represents motion through a sequence of joint coordinates and trained on large corpus of text-motion pairs with classifier-free guidance. We apply the targeting guidance and object avoidance guidance as described (37]. Let (t) denote the joint coordinates at time f, y: the target location, Yobs the obstacle location, the radius of the objects, and T the total number of frames. The loss function is defined as follows:
{ = ||y. (T) + Σ sigmoid(-([<o0() Yobsl r) 50) x 100.
Our experimental configuration adheres to the guidelines set forth in (37] We assess the methods using the targeting loss (the first (8) the object avoidance loss (the second term in (8), and the CLIP score calculated by MotionCLIP [39] In this application, MPGD-Z cannot be applied as there are no auto-encoder. MPGD w/o proj suffers from the shorteut and cannot achieve good performance, as discussed in Appendix B.2 In our method, random augmentation is omitted because the guidance is not computed by neural networks so the adversarial gradient issues are not obvious. The quantitative results of our investigation are summarized in Table 3 while Figure showcaases randomly selected samples. Our methods exhibit enhanced control quality over the generated motion. The videos provided in the supplementary materials.
8

---

"Backwards"
"Balanced Beam"
"Walking"
"Jogging"
Methods
Loss|
CLIPt
Loss
CLIPT
Loss]
CLIP
Loss]
CLIP
Unconditional
40
3.55
9.66
65.6
47.92 0
70.8
48.88
37.6
144.84 0
61.72
FreeDoM 49
1.09
6.63
67.23
9.83 1 4.48
62.65
1.64 + 7.55
40.12
34.95
7.83
58.74
LGD-MC [37
0.98 6.48
67.31
4.42 + 0.02
63.13
1.30
0.39
38.82
6.12 + 2.38
57.89
Ours
0.68+1.32
67.50
1.13+0.30
63.02
0.4340.31
40.40
2.93+1.15
60.03
Table 3: Comparison of various methods on MDM with zero-shot targeting and object avoidance guidance. Loss is reported as a two-component metric: the first part is the MSE between the target and the actual final position of the individual; the second part measures the object avoidance loss.
5.4 Related Work on Training-Fi Guidance
Due to space limitation, we only introduce the related work on training-free guidance while leaving more related work in Appendix A The current training-free guidance strategies for diffusion models can be divided into two primary categories. The first category the loss-based guidance in this paper, which is universally applicable to universal control formats and diffusion models. These methods predict a clean image, subsequently leveraging pretrained networks to guide the diffusion process. Central to this approach are the algorithms based on (5). which have been augmented through techniques like timeetravel [2 49 and the introduction Gaussian noise (7). The adjoint sensitivity accurate guidance gradient. Extensions these algorithms have found utility in domains with constrained data-condition pairs, such molecule generation [14], and in scenarios necessitating zero-shot guidance, like open-ended goals in offline reinforcement learning [44]. In molecular generation and offline reinforcement learning, they outperform training- baaed alternatives additional training presents challenges. This paper delves deeper into the mechanics of this paradigm and introduces suite of enhancements to bolster its performance. The efficacy of our proposed modifications demonstrated across image and motion generation, with promising potential for generalization to molccular modeling and reinforcement learning tasks.
method [31] and spherical Gaussian constraint [47) have been adopted to estimate
The second category of vaining-free guidance is tailored to cext-to-image or text-to-video diffusion models, which is based insights into their internal backbone architecture. For instance, object layout and shape have been linked to the cross-attention mechanisms (17]. while network activations have been shown to preserve object appearance (42] These understandings facilitate targeted editing of object layout and appearance (Diffusion Self-Guidance [T] and enable the imposition of condi- tions ControlNet through training-fre free means (FreeControl [30]). Analyzing these methodologies is challenging due to their reliance on emergent representations during training. Nonetheless, certain principles from this paper remain relevant; for example, as noted in Proposition (3.3] these methods often necessitate extensive diffusion steps, with instances such as [30 MT employing 1000 steps. A thorough examination and refinement of these techniques remain an avenue for future research.
Conclusions
In this paper, conducted comprehensive investigation into training- free guidance, which employs pretrained diffusion models and guides them using the off-the-shelf trained on clean images. Our exploration delved into the underlying mechanisms and fundamental limits of these models. Moreover, we proposed a set of enhancement techniques and verified their effectiveness both theoretically and empirically.
Limitations. Despite our efforts to mitigate the shortcomings of training-free methods and enhance their performance, certain limitations remain. Notably, the refined training-free guidance still necessitates a higher number of NFEs when compared with extensive training methods such as classifier-free free guidance. This is because misaligned gradient cannot be fully eliminated without training.
Ethical Consideration. Similar to other models designed for image creation, our model also has the unfortunate potential to used for creating deceitful or damaging material, We pledge to restrict the usage of our model exclusively to the realm of research to prevent such misuse.

---

The
The RoaNts
Condition
SEACS and
PADIVEL and
UG
LGD-MC
FreeDoM
MPGD
Ours
Figure 4: Qualitative results of CelebA-HQ with shot segmentation, sketch, and text guidance. The images are randomly selected.
A backpack and an elephant
truck below horse cat to the left a A goose the elephant the
UG
LGD-MC
FreeDoM
MPGD
Ours
Figure Qualitative results of ImageNet model with zero-shot text guidance. The images are randomly selected.
The person walking backwards
person walking around in balance beam
The person walking
The person jogging
Uncond itional
MAAM
MAS MAR
FreaDoM
LGD-MC
A x
Ours
Figure 6: Qualitative results of human motion diffusion with zero-shot object avoidance and targeting guidance. Instances of intersection with obstacles are highlighted by marking the person in red. The trajectories randomly selected.
10

---

References
[1] Anish Athalye, Logan Engstrom, Andrew llyas, and Kevin Kwok. Synthesizing robust ad adversarial examples. In International conference on machine learning, pp. 284-293, PMLR, 2018.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843 852, 2023.
[3] Tim Brooks, Bill Peebles. Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Vidco generation models as world simulators. 2024. URL
com/reseacc/
siteoo-gneeraiion models- world- simulators
[4] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: survey, arXiv preprint arXiv:1810.00069, 2018.
[5] Yixuan Chen, Yubin Shi, Mingzhi Dong, Xiaochen Yang. Dongsheng Li, Yujiang Wang. Robert Dick, Qin Lv, Yingying Zhao, Fan Yang, al. Over-parameterizee model optimization with polyae-tojasiewiez condition. 2023.
[6] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances Neural Information Processing Systems, 35: 25683-25696, 2022.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
[9] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
[10] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 1602 1614, 2011.
[11] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36, 2024.
[12] Hyojun Go, Yunsung Lee, Jin- Younoung Kim, Seunghyun Lee, Myeongho Jeong, Hyun Seung Lee, and Seungtack Choi. Towards practical plug-and-play diffusion models. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition. pp. 1962-1971, 2023.
[13] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessiin adversarial examples. arXiv preprint a4Xiv:1412.657) 1412.572, 2014.
[14] Xu Han, Caihua Shan, Yifci Shen, Can Xu, Han Yang, Xiang Li, and Dongsheng Li. Training-fres multi-ohjective diffuusion model for 3d molecule generation. In The Twelfth International Conference on Learning Representations, 2023.
[15] Elad Hazan and Sham Kakade. Revisiting the polyak step size. arXiv preprint arXiv:1905.00312 2019.
[16] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-HSsiang Liao, Yuki Mitsufuji., J Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. arXiv preprint arXiv:2311 16424, 2023.
[17] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.00 01626, 2022.
[18] Jonathan Ho and Tim Salimans. Classifier-frre diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbecl, Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[20] Emiel Hoogeboom. Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference machine learning, pp. 8867-888 8887. PMLR, 2022.
11

---

[21] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffuusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
[22] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-tojasiewiez condition. In Machine Learning and Knowledge Discovery in Databasex: European Conference, FCMII PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16. pp. 795-811. Springer. 2016.
[23] Zhixuan Liang. Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.0 01877, 2023.
[24] Xihui Liu, Dong Huk Park. Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 289-299 2023.
[25] Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang. Hao Su, and Qiang Liu. Fusedream: Training- free text-to-image generation with improved clipt gan space optimization. arXiv preprint arXiv:2112.015733 2021.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast odc solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[27] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu, Contrastive energy prediction for exact energy-guides diffusion sampling in offine reinforcement learning, arXiv preprint arXiv:2304. 12824, 2023.
[28] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using demoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
[29] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1766:::700600083 2017.
[30] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. arXiv preprint arXiv:2312.075366 2023.
[31] Jiachun Pan, Jun Hao Liew, Vincent YF Tan, Jiashi Feng, and Hanshu Yan. Adjointdpm: Adjoint sensitivity method for gradient backpropagation of diffusion probabilistic models. arXiv preprint 2xXvv2307..0711 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Cabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022.
[34] Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019.
[35] Hadi Salman. Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533- 3545, 2020.
[36] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[37] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardami, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. 2023.
[38] Christian Szegedy, Wajciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312,699,,, 2013.
12

---

[39] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference Computer Vision, pp. 358-374. Springer, 2022.
[40] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14911 2022.
[41] Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. Advances in Neural Information Processing Systems, 36, 2024.
[42] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1921-1930, 2023.
[43] Roman Vershynin. High digensiimni probability: An introduction with applications in data science. volume 47. Cambridge university press, 2018.
[44] Wei Wang, Dongqi Han, Xufang Luo, Yifei Shen, Charles Ling, Boyu Wang, and Dongsheng Li. Toward open-ended embodied tasks solving. In NearlPS 2023 Agent Learning in Open-Endedness Workshop, 2023.
[45] Xiaoyu Xiang, Ding Liu, Xiao Yang, Yiheng Zhu, Xiaohui Shen, and Jan P Allebach. Adversarial open domain adaptation for sketch-to- to-ttoto synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022.
[46] Yilun Xu, Mingyang Deng. Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 36, 2024.
[47] Lingxiao Yang, Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusionn arXiv preprint XrXiv:2402.0333 032000 2024.
[48] Changgian Yu, Jingbo Wang. Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network real-time semantic segmentation. Proceedings of the Earopean conference on computer vision (ECCV), pp. 325-341, 2018.
[49] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training free energy guided conditional diffusion model. arXiv preprint arXiv:2303.09M8 2023.
[50] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1836-3847, 2023.
[51] Shengyu Zhao, Zhijian Liu, Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. Advances in neural information processing systems, 33:7559-75700 2020.
[52] Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju. Jiaxi Wang. Jianwei Zhu, Yaosen Min, et al. Towards predicting equilibrium distributions for molecular systems with deep learning. arXiv preprint arXiv:2306.cm 200.:2250000045, 2023.
13

---

A Related Works
A.1 Training-based Gradient Guidance
The training-basedd gradient guidance paradigm, such as classifier guidance, a predominant approach for diffusion guidance. The core objective is to train time-dependent network that approximates Pe(y) in the RHS of (2), and to utilize the resulting gradient as guidance. The most well-known example is classifier guidance, which involves training classifier noisy images. However, classifier guidance limited to class conditions and not adaptable to other forms of control, such as image and text guidance. To address this limitation, there are two main paradigms. The first involves training a time-dependent network that aligns features extracted from both clean and noisy images, as described by [24]. The training process is outlined follows:
min :.) of f (0t, t), fo(o)))
where -) represents a loss function, such as cross-entropy or the &g norm. If dependenn networks for clean images are already available, training can proceed in self- fashion without the need for labeled data. The second paradigm, as outlined by [2I], involves training an energy-based model to approximate yr (y)(z)). The training process is described as follows:
min Bp( (x t), y) - (z)) 3-
However, it is observed in [2 that none of these methods can accurately approximate the true energy in ( The authors of 27 propose an algorithm learn the true energy. The loss function is contrastive loss
exp(e(fo(z(; (),
min
E pp exp(- exp(=
Σ log
Σj=1
(), y"))
where (ai), ax) are K paired data samples from p(arj: y²). Theorem 3.2 22 proves that the
optimal fa- satisfied that Vn, E(Sa= (), = (t.Pi
Although this paper focuses on training-free guidance, the findings in this paper be naturally extended to all training- basen gradient guidance schemes. Firstly, the issue of adversarial gradients cannot be resolved without additional training; hence, all the aforementioned methods are subject to adversarial gradients. Empirical evidence for this is presented in Fig. 2 which illustrates that the gradients from an adversarially robust classifier are markedly more vivid than those from time- dependent classifiers. Consequently, it is anticipated that incorporating additional adversarial training into these methods would enhance the quality of the generated samples. Secondly, since these methods are dependent on gradients, employing more sophisticated gradient solver could further improve their NFEs.
A.2 Adversarial Attack and Robustness
An adversarial attack introduces minimal, yet strategically calculated, changes to the original data that often imperceptible to humans, leading models to make incorrect predictions. The most common attacks gradient-based for example, the Fast Gradient Sign Method (FGSM) [33, Projected Gradient Descent (PGD) [29]. Smoothed Gradient Attacks [1] and Momentum- Based Attacks 9 An attack is akin to classifier guidance training-free guidance, which the gradient of a pre-trained network for guidance. Should the gradient be adversarial, the guidance will be compromised. This paper establishes the relationship between training-free -freinnggreeeeeeee loss-guided diffusion models and adversarial attacks in two ways. Firstly, we prove that training-fre guidance is more sensitive to an adversarial gradient. Secondly, in Section 4.2] we demonstrate that borrowing an adaptive gradient scheduler can improve convergence. The optimizers from adversarial attack literature may also expedite the convergence of the diffusion ODE.
Adversarial attacks and robustness constitute a fundamental topic in deep learning 4
B Baselines and Experimental Settings
B.1 Details of the Baselines
We use the following training- free diffusion guidance methods as baselines for comparison:
14

---

Universal Guidance (UG) [2] employs guidance as delineated in (5) and uses time-travel strategies outlined in Algorithm to enhance performance. The all time steps I. UG also utilizes backward guidance, which takes multiple gradient steps at each time step.
trick is used for
and time-travel trick. In addition, FreeDDMM incor- porates a time-dependent step size for each gradient guidance and judiciously selects the diffusion step for executing time- travel trick.
MreeDoM [9 is also founded
Loss-guided Diffusion with Monte Corlo (LGD-MC) [37 utilizes guidance from 6 and we set n = 10 in the experiments.
Manifold Guided Preserving Diffusion (MPGD) 16 takes the derivative with respect to estimated clean image [[ instead of 2z Let Faj = [ MPGD steps are expressed as following:
Væ, log (y) Va-- V log
MPGD-Z adopts an additional auto-encoder to preserve manifold constraints. The details procedures of MPGD-Z. are described in Algorithm 3 of [6.
B.2 MPGD for Motion Diffusion
For the process of motion diffusion, the application of both MPGD-Z and MPGD-AE is precluded due the absence of pretrained auto-encoders specific to motion diffusion. An implementation of MPGD without projection (MPGD w/o proj) was attempted for motion diffusion; however, it was unsuccessful in accurately navigating towards the target. This failure attributed to the presence of spurious correlations within the targeting loss specific to MPGD, a phenomenon not observed in the other baseline methodologies. Sepcifically, the gradient formulation in MPGD is detailed follows:
{
[[ 2ojz) if = T otherwise
grad MPGD
The gradient in FreeDoM and other methods is given by
grad DPS 2(yuurget Øoje) V2 log (xxe)
Vai
Analysis of the aforementioned equations reveals that, within the MPGD framework, only the final motion step is influenced by the gradient, a characteristic not shared by alternative methodologies. Consequently, this exclusive focus on the last step results in disproportionaely strong guidance at this juncture, while earlier steps suffer from a lack of directional input. This imbalance may adversely affect the overall quality of the samples produced. Empirical observations substantiate that MPGD struggles to achieve targeted outcomes when nuanced adjustment of step size is required. Given these limitations, MPGD been excluded from the comparative analysis presented Table
B.3 Prompts for Motion Diffusion
We follow the prompts and evaluation settings in [37] The prompts are (i) "the person is walking backwards"; (ii) "a person walking around in a balance beam": (iii) "the person is walking"; (iv) "the person is jogging". We consider three different directions for each prompt, and each direction has 10 random seeds, the metrics are then averaged together over the 30 synthesized motions.
More Discussions
C.1
Concentration of Estimated Clean Samples
has been demonstrated in that, given fixed Zp. the distribution of the noisy data æ; is concentrated on spherical shell. An extension of this theorem presented in the "High Dimensional Probability" textbook by Vershynin [43] elucidates that the conditional distribution ( æx)) also exhibits concentration spherical shell, provided that its coordinates are sub- aub-Gaussian.
15

---

Theorem C.1. (Theorem 3.1.1 in (43) Denote Zo = 2, + E R", where ; Assume
that 9i are independent identically distributed, Eig, ri and there exists constant 01 such that
Plg. 2 1) & exp(-cil?), then we have
[ voll2 -r? > S exp(-c2nt"
where is constant.
Theorem C.1 establishes that the distribution ( (z)) exhibits concentration spherical shell an exponential rate. For high-dimensional data, such as images, it is reasonable to infer that Wy is predominantly situated this spherical shell.
C.2 Time-travel Trick
Algorithm 3 Time Travel
1: for tort= = T, do
2:
for 1,... do
3:
#!-1 DDIM with (
4:
if then
5:
8.
01/0t-1, I)
6:
a /Brai-1 + -1-vi11Bppp Ben
7:
end if
8:
xl-1 = zi-1
9:
end for
10: end for
The technique of time-travel, also referred to as "resampling", has been proposed as solution to complex generative problems (28), facilitating successful training- -fren guidance in tasks such as CLIP-guided ImageNet generation and layout guidance for stable diffusion, as illustrated in Figure 2 of and Figure 8 of [2) respectively. The procedure the time-travel trick is shown in Algorithm
g which involves recursive execution of individual sampling steps.
C.3 More Quantitative Experiments
Two-stage convergence: We plot the convergence curve of more images in Figure The diffusion model is ImageNet diffusion and the guidance network is ResNet-50. The convergence two-stage: the objective value oscillates when is large, followed by swift decrease, which verifies Proposition
6.75
6.75
6.75
6.75
Le3s Value
Generated Image
Loss Valtal
Loss Value
Value
6.50
Correct Image Loss
6.50-
6.50
6.50
6.25
6.25
6.25
6.25
6.00
6.00
6.00
6.00
500
500
500
500
Diffusion Time
Diffusion Time
Diffusion Time
Diffusion Time
6.75
6.75
6.75
W
6.75
Loss Value
Loss Value
Loss Value
Loss Value
6.25
6.25
6.25
6.25
500
500
500
5:00
Diffusion Time
Diffusion Time
Diffusion Time
Diffusion Time
Figure 7: More experiments of the convergence plot.
Misaligned gradients: For quantitative experiments, we use robust ResNet-50 to assess the presence of misaligned gradient. Since ResNet-50 serves our guidance network, large loss gap between
16

---

the standard ResNet-50 and the robust variant indicates a severe adversarial gradient issue. In Table the values represent the loss value. The columns denote the guidance network type, while the rows indicate the loss tested on either ResNet-50 or Robust ResNet-50.. The values reported are the average over 1000 images, providing a quantitative evaluation. When using ResNet-50 for guidance, its loss on ResNet-50 is as low as that of real images from the same class (value 5.91). However, the loss for Robust ResNet-50 is significantly higher (value 6. 6.17). suggesting susceptibility to misaligned gradients. By employing ResNet-50 with random augmentation for guidance, we observe marked reduction this gap (from 5.91 to 5.98), underscoring the effectiveness of our proposed method.
Slower Convergence: Table 5 aims to compare the convergence of training- free guidance with training- based guidance. specifically PPAP [12]. Given that classifier guidance presents a simple scenario and all methods converge relatively fast, it tends to obscure the observations. Consequently, we have directed our attention towards the more complex task of segmentation map guidance Table [ The columns in Table 5 list different methods, while the rows indicate the sampling steps. The values represent the "distance" mentioned in T are averaged over 1000 images under the same conditions as those in Table [ values between the training-free FreeDoM and the training-basee PPAP [2). particularly at lower number of steps (20 or 50 steps), which indicates slower convergence. However, when incorporate the Polyak step size into FreeDoM, this convergence issue is substantially mitigated.
Our observations reveal a significant discrepancy in the objective
RN-50
RN-50+RA
Robust RN-50
Real Data
RN-50
5.91
5.91
5.93
5.91
Robust RN-50
6.17
5.98
5.93
5.93
Table 4: Quantitative experiments for the adversarial gradient. RN-50 stands for ResNet-50 and RA stands for random augmentation trick. Robust RN-50 is adversarial robust ResNet-50 from (35) The columns represent different guidance networks.
FreeDoM
FreeDoM + P
PPAP (Training-basel)
DDIM-20
2439
2190
2032
DDIM-50
1821
1635
1607
DDIM- 100
1657
1504
1509
Table 5: Quantitative experiments for the slower convergence. stands for Polyak step size. The experimental setting follows the segmentation map guidance of Table
C.4 Efficiency of Random Augmentation
Given the multiple invocations of the guidance network necessitated by random augmentation (RA), concerns regarding the efficiency of this approach are understandable. However, is important to note that, compared to the diffusion backbone, the guidance network exhibits more lightweight architecture, thereby mitigating any significant increase in computational demand. To empirically illustrate this point, we present the computation times associated with varying degrees of augmentation in Table 6 In the conducted experiments, we set the cardinality of the set T to 10, thereby having little impact the inference time. These experiments were conducted on single NVIDIA A100 GPU.
Setting
w/o RA
|T| =
IT| = 10
|T]
20
|T| 30
CLIP Score
0.541
0.544
0.571
0.592
0.625
Table 6: Inference time (seconds per diffusion step) of different random augmentation configurations. The diffusion backbone is ImageNet diffusion and the guidance network is CI.IP-/16.
17

---

D Proofs
D.1 Definitions
This subsection introduces a few definitions that are useful in the following sections.
Definition D.1. (L-Lipschitz) A function f R R said to be L-Lipschitz there exists
constant L 2 0 such that (x2) - f(z1)ll I L||æ - æ1 for all œ1, 22 E IR
Definition D.2. (PL condition) function f IR" - R satisfies PL condition with parameter j [[V[()]] 2 (z))..
D.2
Proof for Proposition 3.1
Denote @o (p( (z)). the gradient guidance term (5) can be written the following:
de
E. of lo (2)) ae Cov [
Vz, (f. y)
dāo Zx
(9)
Vat
Eæo of vav
where the last equality follows the variance of Tweedie's formula Cov (2ng ard] ofv² log Pt())) [10].
of
For the first condition, the Lipschitz constant satisfies
Lf
- - (x22) <
2 Va, logpr(<<)) 02 + Va, log (xgp(()))) (2x2)
Var
L,
+ Lp)
x0 x211.
att
and the PL constant satisfies
1 0e
1 VV e Cov
of μ
(xx) S 11
Cov (x0l=x))))
V a. elp
min
The second and third conditions directly follow Lemma D.3
Lemma D.3. (Linear Convergence Under PL condition; (221) Denote x. as the initial point and x as the point after t gradient steps. If the function is L-Lipschitz and p-PL, gradient descent with step size 1) converges to a global solution with E(x') (1 ((xx)
D.3 Proof for Proposition 3.2
Proof. The proof this theorem is based on the proof Lemma expectation, have
34
By the definition of
f(x) = EE + toteel
(f @ N0 ())
(2no7)n/2 S f(z) exp
1
|
21|²
dz.
20
We then show that for any unit direction u- vi(x) S
2
The derivative of f is given by
(-
1
f((x) m//2
f(z)
||
x||2
dz
20
1
f(=) z) exp
|| =1(2 dz.
(2not)m/20f JR.
20?
Thus, the Lipschitz constant computed as
1
2 Tvf()) <
lu Tx 2)/00 10//0/es0 exp
x 2||2
dz
(2mof)a/22
20²
R"
C
[1] --
2
|s| c/eep
ds
1/7
πo²
18

---

Similarly, for the Lipschitz constant of the gradient, we have
|| lao)llop = [|V2f(x)
2TTO of ( 3² expl
1 20 /////d + / of exp((=
20
2
02
D.4 Proof for Proposition 3.3
of single step from time We denote the update variable for DDIM is z; and the optimal solution the diffusion ODE at time as æ. Let at vi 0t A 2 log( aa. and h X Xs. According to (B.4) of (26), the update of DDIM
Proof. We first analyze the discretization
solver is given by
Cle a, x. or(e" - 1)u( 8).
(10)
similar relationship can be obtained for the optimal solution
at x
ac(e" s)
(11)
Oxs
We then bound the enror between and a.. We have
| - xell
x - x,ll - +
ozle" - 1)]lu(x" s) u(x,, s)1 + 0(h²)
a2
| x.1l + o.(6" 1)T,1 x.11 0(22)
O,
S O(11 D)[ x=,1) O
If we run DDIM for M steps, Imax = O(1/M), and we achieve the discretization error bound for
DDIM algorithm:
a00 = O(MM1 M) z + +0((1 L.M
D.5
Proof for Proposition 4.1
Proof.
By the definition of expectation, we have
f(x) Ee- If(= o.c) ())) / f(z)p(æ 2)d2.
Then we compute the Lipschitz constant
V')(()) C
/ |Vp(æ [Vp(x- 2)llad - |VD(())l2dz.
gm
As for the gradient Lipschitz constant, we have
L = |V [<>)(>) s c c V2p(æ 2)d2
sC [g VVp(())lapdz,
where || llop is the operator norm of a matrix.
E More Qualitative Results
E.1 CelebA-HQ
E.2 ImageNet
19

---

Condition
UG
LGD-MC
FreeDoM
MPGD
Ours
Condition
UG
LGD-MC
FreeDoM
MPGD
Ours
Figure 8: More qualitative results of CelebA-HQ with zero-shot segmentation guidance. The images are randomly selected.
E.3
Human Motion
See "TeX Source/videos.pptx..
20

---

Condition
UG
LGD-MC
FreeDoM
MPGD
Ours
Condition
UG
LGD-MC
FreeDoM
MPGD
Ours
Figure 9: More qualitative results of CelebA-HQ with zero-shot sketch guidance. The images are randomly selected.
21

---

Condition
IMPOL
reeding
METISIAA
UG
LGD-MC
FreeDoM
MPGD
Ours
SANETTO OPEE Conreataeee
Condition
UG
LGD-MC
FreeDoM
MPGD
Ours
Figure 10: More qualitative results of CelebA-HQ with zero-shot text guidance. The images are randomly selected.
22

---

A castle in the sky
A church under moonlight
A fireboat
UG
FreeDoM
MPGD
Ours
A flaming robot walking on ice
Dog looking camera
Sunshine
LGD-MC
FreeDoM
MPGD
The city underwater
The traffic on a commercial stret
Windows XP background
UG
LGD-MC
FreeDoM
MPGD
Figure 11: More qualitative results of ImageNet with zero-shot text guidance. The images are randomly selected.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that the main claims made in the abstract and introduction accu- rately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the conclusions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
The assumptions and proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper details the evaluation benchmarks and baselines, while the code made available in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
The source code is provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
seipaiisssooooiioyyo. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our experiments are designed in alignment with established studies, which guided the setup of these components.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In the experiments, we adhere to methodologies outlined in existing studies, which do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on a single 100 GPU, although less powerful GPUs, such as the V100 with 16GB of memory, are sufficient for replicating the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph::///// httppp//////uiss...ee /ttngggttrr///prrrrp...c.///a/tt
Question:
Answer:
[Yes]
Justification:
We confirm that the research conducted the paper conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The societal impacts are discussed in the conclusions.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: All checkpoints referenced are documented in existing literature, which did not specify any safeguards. Consequently, it is challenging to implement safeguards for these checkpoints.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "pLs6gmYNri", "reviewer": "Reviewer_wPsc", "summary": "The paper studies training-free loss-based diffusion guidance. in comparison to classifier-based guidance. First, the paper examines several drawbacks of loss-based guidance, including (1) while successful minimization of the loss can be achieved, it does not guarantee successful guidance, (2) loss-based gradients can be misaligned with the desired guidance, in contrast to the time-dependent classifier-based gradients with improved smoothness properties, (3) the superior smoothness of classifier-based guidance also means that loss-based guidance takes longer to converge, incurring more function estimations.\n\nIn response to those drawbacks, the authors propose two techniques to improve loss-based guidance: (1) random augmentations to improve its smoothness, and (2) Polyak step size to further improve robustness to misalignment between initialization and the specified condition for generation. Three applications are considered for evaluation, where the proposed technique is implemented on top of FreeDoM.", "questions": "Technical:\n========\n- Section 3.2 and elsewhere:\n  - Recommend to use terms along the lines of \"misaligned gradients\" rather than \"adversarial gradient.\"\n    - I can see it helps to leverage earlier insights from adversarial robustness, e.g., regarding the role of Lipschitz constants, but I worry that there's no benefit to confusing the behavior in question with adversarial robustness considerations. If anything, those gradients were not chosen to maximize deviations w.r.t. any particular objective, but just happen to be misaligned.\n    - This suggestion seems to match the narrative on L166-168.\n    - Not sure if *adversarial* on L258 is related to this discussion, but just pointing it out.\n  - I'm not sure what's exactly meant by \"non-Lipschitz\" functions\n    - A function that's, say, 200-Lipschitz or only 2-Lipschitz is still not 1-Lipschitz.\n    - It seems that Proposition 3.2 only needs the upper bound on $\\ell$, but need not make any statements about its smoothness.\n    - Similarly, L149 can simply state that adding Guassian noise improves smoothness, which is pretty intuitive.\n    - It would help to double-check if Proposition 3.2 follows from known standard results, e.g., randomized smoothing which seems to be what $\\hat{\\ell}$ is about. See Assumption A in\n      - Duchi, John C., Peter L. Bartlett, and Martin J. Wainwright. \"Randomized smoothing for stochastic optimization.\" SIAM Journal on Optimization 22, no. 2 (2012): 674-701.\n    - Same goes for Proposition 4.1.\n    - is it true that Proposition 3.2 is a special case of Proposition 4.1?\n- L253: Is there a way to quantify this mismatch between the condition and the input?\n- I can't readily place where time travel factors into the presentation and/or the experiments. Is it only mentioned in passing? Is the related analysis in the appendix only included as a bonus?\n\nPresentation:\n==========\nSection 1:\n  - L43: Please cite Appendix E in [25] specifically, since the paper may otherwise seem to be concerned with a different problem altogether, i.e., offline reinforcement learning.\nSection 2:\n  - Suggest to flatten the section by using bold headers as used later. (Usually looks better than an absent opening paragraph.)\n  - Section 2.1 could use some citations, also to clarify which formulation or model is adopted in this paper, and whether the contributions apply to other models as well.\n  - L84: Please fix `fastRCNN` and include a citation.\nSection 3:\n  - Proposition 3.1\n    - Recommend to at least provide an intuitive definition of PL. Otherwise, I'd recommend to present an *(informal)* version of the statement that's optimized for readability, deferring the formal version in its entirety to the appendix.\n  - Figure 1: would help to explain that a wrong image was obtained in (b).\n  - L132: Seems like a good place to start a new paragraph at \"Furthermore\"\n  - L172: Recommend to cite a textbook.\n  - L179: Promposition 3.3\n    - Is it necessary to reuse the symbol $g$?\n    - $g(x_t, t)$ is L-Lipschitz w.r.t. which parameter?\n    - It is not clear how the discretization error is defined here. Some context would help.\n    - Is it necessary to use the symbol $h_{\\text{max}}$? It seems to have nothing to do with the function $h(x_t, t)$.\n    - Is $O(h_{\\text{max}} + Lh_{\\text{max}})$ the same as $O((L+1)h_{\\text{max}})$?\n    - After a second reading, it's still not clear to me how the discretization error relates to the rate of convergence.\nSection 4:\n  - Algorithm 1 & 2: Suggest to highlight Tweedie's formula\n  - L189: Please include a citation for the Polyak step size.\n  - L191: Is it really the case that the analysis presented \"completes the picture\" for improving training-free guidance? If so, it would help to describe what it takes to complete the picture, then explain how the analysis answers to that. It would also help to highlight any remaining gaps or extensions for future work.\n  - L206: Recommend to give names to those integrations defining the Lipschitz bounds.\n\nNitpicking:\n========\n- Recommend to reduce redundancy:\n  - L1: Adding additional control -> Adding / Controlling / Guiding\n  - L19: the generation / the synthesis, as well as the creation -> generation\n  - L32: zero-shot generalization to novel conditions -> zero-shot generalization / generalization to novel conditions\n  - L38: employ pre-trained networks, designed for clean images [..] checkpoints for these networks pretrained on clean images are -> employ networks pretrained on clean images [..] pretrained networks are\n- Recommend to improve consistency in terminology and notation:\n  - pre-trained and pretrained\n  - clarifies the mystery and resolves the mystery\n- Suggest to favor technical terms:\n  - \"mystery\" of guidance weights -> role of guidance weights, principles for choosing guidance weights, intricate interplay between guidance weights and the guidance function and time.\n  - L190-191: \"trick\" -> approach / algorithm / heuristic\n- L99: Despite its intuitive -> Despite being intuitive\n- L100: considers -> consider\n- L100: with Gaussian dist. -> with a Gaussian dist.\n- L102: for one-dimension dist. -> for a one-dimensional dist.\n- L130: referred as -> referred to as\n- L195: incorporating them into -> passing them to\n- L212: emerges -> emerge\n- L213: efficiency -> efficacy?", "limitations": "Although the authors seem to have aimed to complete the picture for training-free loss-based guidance, covering both its drawbacks and potential improvement strategies, this picture is still not clear to me.\n\nFor example, it's not clear what's best possible under reasonable conditions on the loss employed for guidance, e.g., how it compares to the (time-dependent) classifier models that can otherwise be employed. I suspect this has something to do with the limitations of time travel. It would be nice to attempt to show some lower-bounds on the errors incurred through time travel.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Offers theoretical results explaining the mechanisms of loss-based guidance\n- Demonstrates the drawbacks of loss-based guidance, in terms of potential misalignments between its gradients and desired guidance directions, and how the smoothness properties of the employed guidance impact the rate of convergence.\n- Proposes two techniques to ameliorate loss-based guidance, as demonstrated on a number of applications.", "weaknesses": "Nothing major, though the presentation of Section 3 leaves something to be desired. I'd need to see it after some polishing before I can be satisfied that I fully understand what's going on. See the comments below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful comments and recognition of this work, especially acknowledging our theoretical contributions.\n\n> Misaligned gradient.\n\nThank you for suggesting a better name. We agree that the misaligned better captures the behavior and will modify it in the later versions of the manuscript.\n\n> I'm not sure ... which is pretty intuitive.\n\nWe agree that the statement pointed out by the reviewer is more appropriate and will change the manuscript accordingly. \n\n> Proposition 3.2 and 4.1 and mentioned reference.\n\nWe thank the reviewer for pointing out this important reference. Proposition 3.2 is a special case of Proposition 4.1 and Proposition 4.1 follows the proofs in Appendix E of the mentioned reference. We will discuss this reference in the revised manuscript. \n\n> The role of time-travel section in the appendix.\n\nTime travel is a widely adopted trick for training-free guidance, but there is no theoretical justification. In the baselines, UG, FreeDoM, and MPGD all adopted it. As a result, we include the theoretical analysis in the appendix to support when it works. \n\n> $g(x_t,t)$ is L-Lipschitz w.r.t. which parameter?\n\nWe assume that $g(x_t,t)$ is L-Lipschitz w.r.t. its first parameter.\n\n> How the discretization error relates to the rate of convergence\n\nWe apologize if our initial description of the discretization error was not sufficiently clear, potentially leading to confusion. To clarify, the discretization error mentioned in Proposition 3.3 refers to the discrepancy between the solution provided by DDIM and the optimal solution. In the context of diffusion, the maximum step size, denoted as $h_{\\max}$, is inversely proportional to the number of steps taken. Consequently, the convergence rate can be expressed as $O((L+1)/T)$.\n\n> Is it really the case that the analysis presented \"completes the picture\" for improving training-free guidance?\n\nWe apologize for any confusion caused by our initial explanation. In our paper, we provide theoretical justifications for the techniques used in existing training-free guidance literature, such as step size adjustments and 'time-travel.' Additionally, we have identified and analyzed key limitations. Our work aims to offer a more comprehensive understanding of the current training-free guidance approaches. We acknowledge, however, that the strategies for addressing misaligned gradients and enhancing convergence is not optimal require further development. We will ensure to clarify these points in the revised manuscript.\n\n> Presentation\n\nWe sincerely thank the recommended changes. We will change the manuscript accordingly.\n\n> Nitpickings\n\nWe sincerely thank the reviewer for the careful reading of the paper. We will correct these typos accordingly and have a careful check in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ySrISf7g7n", "reviewer": "Reviewer_P7k4", "summary": "This paper examines the mechanisms and limitations of training-free guidance for diffusion models and develops a collection of techniques to overcome the limitations accompanied by both theoretical and empirical results.", "questions": "1.\tHow is the performance of MPGD and LGD-MC modified with the random augmentation and Polyak step size?\n2.\tHow is the proposed method compared with DPS, DSG and RED-diff?", "limitations": "The authors have clearly presented the limitations in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper performs the theoretical analysis on the training-free diffusion model from the optimization perspective, which presents the problems and drawbacks in training-free guidance.\n2.\tThis paper also proposes corresponding solutions for the two issues in training-free guidance: random augmentation for the adversarial gradient, Polyak step size for improving the convergence.", "weaknesses": "1.\tThe paper misses the recent literature in training-free diffusion model, such as RED-diff [1] and DSG [2].\n2.\tAlthough the theoretical analysis in this paper illustrates the problems in training-free guidance, the proposed solutions are not particularly novel. Similar ideas have been applied in previous works such as LGD-MC and DSG.\n\n[1] Mardani M, Song J, Kautz J, et al. A variational perspective on solving inverse problems with diffusion models[J]. arXiv preprint arXiv:2305.04391, 2023.\n\n[2] Yang L, Ding S, Cai Y, et al. Guidance with spherical gaussian constraint for conditional diffusion[J]. arXiv preprint arXiv:2402.03201, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedbacks. We have added the baselines and integrated the proposed methods into MPGD and LGD-MC.\n\n> Question 1: MPGD and LGD-MC with random augmentation and Polyak step size\n\nWe are grateful for your suggestions to enhance our experimental section. As requested, **we have included additional experiments in Table 1 in PDF file of the global rebuttal**. In these experiments, when integrating LGD-MC, we set the number of random augmentations to 5 and the Monte Carlo iterations to 2. The results show that both LGD-MC and MPGD significantly outperform their original counterparts when combined with random augmentation and Polyak step size adjustments. This indicates that our proposed methods can complement existing techniques. Through further analysis, we determined that the Polyak step size is the primary factor contributing to these improvements. Notably, MPGD-Z, when used in conjunction with our approach, delivers exceptional performance, even surpassing the training-based PPAP [B].\n\n> Weakness 1 and Question 2: Comparison with DPS, DSG, and RED-diff\n\nThank you for highlighting the necessity of including additional baselines. We have conducted further experiments with the DSG algorithm and presented the results in Table 1. It is important to clarify that the DPS algorithm is essentially the FreeDoM method, minus the time-travel techniques. Since DPS is analogous to FreeDoM without the time-travel aspect, we believe that the performance of FreeDoM can be indicative of DPS's performance. As for RED-diff, it employs diffusion as a form of regularization for the least squares problem. It is more suitable for linear inverse problems rather than conditional generation tasks. In our testing, RED-diff failed to generate valid images under conditions such as segmentation, sketch, or CLIP guidance. Consequently, we have decided not to include RED-diff in our baseline comparisons.  **We have incorporated the DSG results into Table 1 in the PDF file of the global rebuttal**. While DSG shows an improvement over FreeDoM, it still falls short when compared to our proposed methods. \n\n> Weakness 2: Proposed solution is not novel compared with LGD-MC and DSG. \n\nWe appreciate your assistance in elucidating the distinct contributions of our work. The approaches of LGD-MC and DSG differ in their motivations and are orthogonal in their methodologies. In LGD-MC, the MC method is utilized to approximate the posterior distribution, which necessitates that the variance of the Gaussian noise be proportional to $\\sigma_t/\\sqrt{1 + \\sigma_t^2}$. On the other hand, the rationale behind employing random augmentation is to mitigate adversarial gradients, allowing for the consistent application of the same augmentation across different steps. It is worth noting that random augmentation and MC can be employed concurrently, although this may result in an increased sampling time. The essence of DSG lies in the projection onto the manifold, a process that is orthogonal to the choice of step size. Moreover, various manifold projections can be seamlessly integrated with the Polyak step size, as detailed in [A]. \n\n[A] Iteration-complexity of the subgradient method on Riemannian manifolds with lower bounded curvature. Optimization 2019.\n\n[B] Towards Practical Plug-and-Play Diffusion Models. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFgLKOLrM6", "reviewer": "Reviewer_DfoY", "summary": "This paper explores the theoretical aspects of training-free loss-based guidance mechanisms in diffusion models and improves them from theoretical findings. The first results are in explaining why the guidance strength depended on $\\sqrt{\\alpha_t}$ worked well in FREEDOM. Then, the authors explore the adversarial gradients, which result in a lower loss but produce generations with the incorrect condition. Specifically, they demonstrated that a time-dependently trained network with Gaussian noises is smoother compared to a general network. This smoothness can cause a slowdown of guidance. From these theoretical results, they propose to utilize random augmentation to increase the smoothness of the off-the-shelf network and improve the convergence speed with Polyak step size, which accelerates convergence by adaptive gradient step size. Experimental results show that their method can outperform loss-based guidance methods including Universal Guidance, Loss-Guided Diffusion, FreeDom, and MPGD-Z.", "questions": "## Typo\n- Line 875 in Appendix: $||f(x_2)-f(x_2)||$ ->  $||f(x_2)-f(x_1)||$\n- Equations are referred to as (number). I am very confused to understand whether (number) is an equation or a bullet point. How about using Eq. (number)?\n\n\n## References\n\n- [A] ELUCIDATING THE DESIGN SPACE OF CLASSIFIER GUIDED DIFFUSION GENERATION, ICLR 2024\n\n- [B] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.\n\n- [C] ADJOINTDPM: ADJOINT SENSITIVITY METHOD FOR GRADIENT BACKPROPAGATION OF DIFFUSION PROBABILISTIC MODELS, ICLR 2024.\n\n- [D] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method, Arxiv 2023.", "limitations": "The higher number of NFEs required in training-free guidance and adversarial guidance are mentioned in the limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper explores the theoretical aspects of training-free loss-based guidance. The authors explain well-known practices in previous works with theoretical aspects, such as FreeDom.\n- Various benchmark datasets are utilized for evaluating their method.", "weaknesses": "- My first major concern is missed baselines and related works. This work aims to improve off-the-shelf models’ guidance with the training-free based method, and [A] deals with the same configuration. [A] elucidates the design space of off-the-shelf guidance, and smoothed classifier, joint classifier guidance, and guidance schedules are explored in their work. I think that the smoothed classifier concept in [A] is very similar to random augmentation for the smoother network in this work, and similar guidance schedule schemes are explored in [A] and this work. Therefore, [A] should be discussed in this work and be compared in the experimental section. Also, I found [C] and [D].\n- In the other line of work, [B] proposes the utilization of off-the-shelf models by finetuning that model to be the time-dependent network. Although [B] can be categorized into the training-required method, it would be better to discuss this kind of training-required method and compare it in the experimental section. Showing the performance gaps between training-free and training-needed methods is needed to validate the effectiveness of the proposed method.\n- [A] and [B] choose the imagenet classifier guidance with imagenet trained diffusion models and various diffusion models use imagenet class conditional generation as the core benchmark. However, this work does not incorporate this major benchmark in their benchmark, so it can make reader hard to recognize their absolute performance.\n- There are no details about the Polyak Step size, and if Algorithm 2 is all of the details of the Polyak Step Size, there have been many works to utilize normalized gradients and I felt that this is not novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedbacks. Based on the comments, we have added more baselines. \n\n> Weakness 1 and 2: Comparison with [A,B,C,D]\n\nWe appreciate the reviewers for highlighting these pertinent references. We have now included [B] (training-based PPAP) and [D] (SAG) as additional baselines, with the results presented in Table 1 of the PDF for global rebuttal. Regarding hyperparameters, we have opted for a solve step of $n=4$ and selected the number of experts in PPAP to be $10$, consistent with the original papers.\n\n[A] primarily focuses on employing a classifier as the guidance network. For instance, the adoption of a joint direction, a softplus classifier, and the adjustment of classifiers' temperature are approaches that are difficult to be applied to general loss functions. Since [D] builds upon the general methodology outlined in [C], we have chosen to include only [D] in our analysis. Our findings indicate that SAG outperforms FreeDoM, particularly in terms of the objective value, aligning with the results reported in the SAG paper. However, it is surpassed by our proposed method. Moreover, the performance of our method is comparable to that of training-based approaches.\n\nWe would also like to clarify the differences between our work and [A]. Firstly, regarding the smooth classifier, the methodology employed in [A] involves using a softplus classifier and adjusting the classifier's temperature, which may not be readily adaptable for general conditions. In contrast, the random augmentation method we propose is independent of the loss function and guidance network. From a motivational standpoint, our paper introduces an additional significant motivation: to eliminate the adversarial gradient, an aspect that has been overlooked by existing studies. Secondly, concerning the scheduler, our paper proposes a step size choice that is orthogonal to the scheduler design.\n\n> Weakness 3: Class condition as benchmark\n\nSimilar to FreeDoM and LGD-MC, our work concentrates on tasks for which training-based methods are not yet fully developed. We have utilized the same benchmarks as those employed in FreeDoM and LGD-MC. Given that both FreeDoM and LGD-MC have demonstrated commendable performance on these benchmarks, we believe that surpassing these established baselines is indicative of our method's robust performance. We are open to different viewpoints and welcome the opportunity for further discussion.\n\n> Weakness 4: Polyak Step size\n\nWe are grateful for the opportunity to clarify the novelty of our approach. Due to space constraints in the paper, we were unable to delve into the specifics of the Polyak step size, which may have led to some misunderstanding. The Polyak step size is a method of choosing step sizes, distinct from normalization, and it unifies various step size choices with a proven optimal convergence rate under a range of conditions [13]. In contrast, normalization methods like those used in other works (e.g., DSG) typically focus on projection to the manifold. It's important to note that the Polyak step size and manifold projections are orthogonal concepts [E], and their combination in our method is more than a mere amalgamation of the two methods' codes. Furthermore, we have compared our method against the state-of-the-art DSG [F], as shown in the global response PDF, and our proposed method demonstrates superior performance.\n\n> Typo\n\nThank you for your careful reading. We will carefully correct the typos in the revised manuscripts.\n\n[E] Iteration-complexity of the subgradient method on Riemannian manifolds with lower bounded curvature. Optimization 2019.\n\n[F] Guidance with spherical gaussian constraint for conditional diffusion. arXiv:2402.03201.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JX03z38UOg", "reviewer": "Reviewer_uPWj", "summary": "This paper explores training-free loss-based diffusion guidance, providing an overview of how training-free guidance functions from an optimization perspective. It analyzes the challenges of training-free guidance, particularly regarding adversarial gradient issues and slower convergence rates. Additionally, the paper proposes two improvements: random augmentation and Polyak step size. These enhancements are validated through experiments across various applications.", "questions": "- What is the rationale behind choosing the Polyak step size? It appears to be a post-hoc decision without strong intuition.\n- Since random augmentation is omitted in MDM, is the only difference between “Ours” and “FreeDoM” the Polyak step size? The loss values in Table 3 show a significant difference between FreeDoM and Ours, and clarification on this point is needed.\n- What's the value of $\\eta$ set for DDIM?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The theoretical analysis offers valuable insights into understanding training-free guidance.\n\n- The proposed tricks, random augmentation and Polyak step size, demonstrate effectiveness.", "weaknesses": "Although the paper provides a thorough theoretical analysis of why training-free guidance works and doesn’t work, the empirical evidence is insufficient to fully support the claims. The propositions are based on the assumption of Lipschitz continuity, which may not hold for complex deep neural networks, therefore, a more comprehensive quantitative analysis is necessary. Here are my detailed comments:\n\n- The experiment presented in Figure 1 is somewhat confusing. The authors aim to convey three points: (1) the loss curve will oscillate initially and decrease eventually; (2) the guidance weight needs careful selection; (3) adversarial gradients may cause incorrect generated images despite low loss. The first point requires more extensive experiments rather than just two cases. The second point is not clearly inferred from the figure.\n- The two limitations of training-free guidance, adversarial gradient, and slower convergence rates, lack quantitative experimental evidence.\n\nRegarding the proposed method, while benchmarking many baselines on various tasks is appreciated, several issues remain unaddressed:\n\n- The difference between UG and FreeDoM in this paper appears to be the selection of diffusion steps for the time-travel trick. However, UG also uses a “backward universal guidance” technique as mentioned in section 3.2 of the original paper [1].\n- The authors did not choose classifier-based tasks, which is curious given that the paper analyzes the inferiority of training-free guidance compared to training-based guidance through this kind of task in Sec 3. Comparative experiments showing the gap between these approaches are essential. Including classifier-guided generation tasks would be beneficial to assess the significance of the two tricks’ improvements, and training-based baselines can be included in this task.\n- The selection of hyper-parameters is concerning. With numerous hyper-parameters involved, how are the optimal values (e.g., step size, LGD-MC Monte Carlo samples, diffusion step range for time-travel) chosen? How is it ensured that baselines are implemented optimally?\n- The random augmentation trick may have limitations:\n  - While image augmentation techniques are mature, effective augmentation tricks for other domains (e.g., text, AI4Science, offline RL) may not be available, limiting the trick’s application.\n  - Despite the claim that random augmentation does not introduce significant computational overhead, this may not hold true when using a large guidance network, as would be the case with a large, generalist foundation model for scoring samples.\n\n\n\n[1] UNIVERSAL GUIDANCE FOR DIFFUSION MODELS. https://openreview.net/pdf?id=pzpWBbnwiJ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedbacks. We have added more experiments, baselines, and ablation studies. \n\n> Weakness 1: Assumption of Lipschitz continuity\n\nThe assumption of Lipschitz continuity is a standard prerequisite for analyzing diffusion models (e.g., [A]). Without this assumption, constructing a similar argument becomes significantly more challenging (comparing [A] to [B]). Additionally, this assumption is frequently utilized in the analysis of neural networks (e.g., [C]). Therefore, we have incorporated this assumption to facilitate deeper insights while maintaining a streamlined proof structure. \n\n> Weakness 2: The experiment presented in Figure 1 is somewhat confusing.\n\nThank you for your assistance in clarifying Proposition 3.1 and Figure 1. The outcomes (1) and (2) are direct consequences of Proposition 3.1, and Figure 1 serves merely as an illustration of the first claim. We acknowledge the reviewer's point that verifying the first claim necessitates more experiments, and we have included **additional experiments in Figure 1 within the PDF of the global rebuttal**. Claim (3) is discussed in Section 3.2, and we will ensure to revise this paragraph in the updated manuscript accordingly.\n\n> Weakness 3: Quantitative experimental evidence of two drawbacks.\n\nThe qualitative evidence supporting the presence of adversarial gradients and slower convergence rates is presented in Figures 2 and 3, respectively. We concur with the reviewer on the importance of quantitative evidence and **we include quantitative experiments in Table 3, 4 in PDF of global rebuttal**. To evaluate adversarial gradients, we employ a classifier setting with pre-trained adversarially robust networks and test on 1,000 generated images. When utilizing ResNet-50 as the guidance network, it minimizes its own loss, yet the loss on a Robust ResNet-50 remains high, indicating the existence of adversarial gradients. Our proposed random augmentation method is effective in mitigating these adversarial gradients.\n\nRegarding the slower convergence, the simplicity of the classifier setting may obscure this observation. Therefore, we shifted our focus to the more challenging segmentation map guidance. Here, we observed a noticeable gap in the objective values between the training-free FreeDoM and the training-based PPAP [D] when the number of steps is small, demonstrating the slower convergence. The implementation of the Polyak step size helps to narrow this gap.\n\n\n> Weakness 4: Description of UG\n\nWe thank the reviewer for correcting this point and we will modify the description of UG in Appendix.\n\n> Weakness 5: Classifier-based tasks and ablation studies\n\nThank you for your valuable suggestion. Similar to FreeDoM and LGD-MC, our work is concentrated on tasks where training-based methods have yet to be fully developed. Given that both FreeDoM and LGD-MC have demonstrated commendable performance on these benchmarks, we believe that surpassing these established baselines is indicative of our method's strong performance.\n\nWe concur with the reviewer on the importance of incorporating training-based methods and conducting ablation studies. To address this, we have included training-based methods and an ablation study of our approach in **Table 2 of the PDF in the global rebuttal**. Polyak step size expedites convergence and significantly enhances the objective value. While random augmentation does improve the Fidelity (FID), it can adversely affect the objective value due to the introduction of randomness. When these two elements are combined, they yield the best performance.\n\n> Weakness 6: Hyperparameter choice\n\nRegarding the step size selection, we initially explored a range from $10^{-5}$ to $10^3$. If a particular step size $\\lambda$ yields valid images, we then refine our search within the interval [$\\lambda$, $10\\lambda$] using 20 evenly spaced values. In the case of Monte Carlo samples, the LGD-MC study noted negligible differences between using $n=10$ and $n=100$, and thus we adopt $n=10$. We sourced the implementations of UG, FreeDoM, and MPGD directly from their respective GitHub repositories.\n\n> Weakness 7: Limitations of random augmentation\n\nRegarding the first limitation, there exist data augmentation methods within the domains of molecular [A] and offline reinforcement learning [B]. For the second limitation, we concur with the reviewer that the adoption of a foundation model may result in increased computation time. However, it is important to note that random augmentation processes are highly parallelizable, and the guidance models utilized for current training-free guidance are not so large. We have provided detailed timings for the CLIP model in Table 4 of the original paper.\n\n> Question 1: Rationale behind Polyak\n\nTraining-free guidance is indeed challenged by a slow convergence issue, as outlined in Proposition 3.3. The simplest remedy for this is to adjust the step size. The Polyak step size has been demonstrated to be optimal under a variety of conditions [13], which is why we have chosen to implement it in our approach.\n\n> Question 2: Clarification on MDM\n\nUnlike image generation, motion diffusion involves creating $192$ frames of motion where consistency between neighboring frames is crucial to produce a viable trajectory. This requirement significantly complicates the optimization challenge, making the selection of an appropriate step size even more critical. Due to this complexity, MPGD struggles to generate a feasible motion, as detailed in Appendix B.2.\n\n> Question 3: the value of $\\eta$\n\nThe value of $\\eta$ is set the same as the released code of these baselines.\n\n\n[A] AugLiChem: data augmentation library of chemical structures for machine learning. MLST 2022.\n\n[B] Reinforcement learning with augmented data. NeurIPS 2020. \n\n[C] On the spectral bias of neural networks. ICML 2019. \n\n[D] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

We would like to express our sincere gratitude to all the reviewers for their constructive feedback and recognition of our work. We are particularly grateful for the acknowledgment of the theoretical contributions (Reviewer wPsc, Reviewer P7k4, and Reviewer DfoY), the novelty of the identified drawbacks for existing training-free guidance (Reviewer wPsc).  

First, we would like to re-emphasize the novelty and technical contributions of this work:

- Our work revisits the concept of training-free guidance from an optimization perspective, which is a novel approach that yields several insights: (1) it provides the first guarantee that generated samples from training-free guidance will have a low guidance loss (Proposition 3.1); (2) it is the first to demonstrate that training-free guidance can suffer from adversarial gradients and slow convergence (Propositions 3.2 and 3.3); (3) it offers the first theoretical support for 'time-travel', a commonly employed technique in baseline models (Proposition C.3).


- We introduce random augmentation and the Polyak step size as solutions to mitigate the issues of adversarial gradients and slow convergence, respectively. These solutions are orthogonal to previous work, allowing them to be integrated with methods such as FreeDoM, MPGD, and LGD-MC to enhance performance. Our proposed solutions have shown significant improvements over these methods and even surpass some training-based approaches.


Based on reviewers' constructive feedback, we have added more baselines and ablation studies. The main points of our rebuttal include:
- We have included additional training-free methods, DSG [A] and SAG [B], as well as the training-based method PPAP [C], in Table 1 of the PDF, addressing Weakness 1 and 2 mentioned by Reviewer DfoY and Weakness 1 by Reviewer P7k4.
- We have applied the proposed techniques to LGC-MC and MPGD-Z to demonstrate their effectiveness, as shown in Table 1 of the PDF, in response to Question 1 from Reviewer P7k4.
- We have conducted qualitative experiments to illustrate the issues of adversarial gradients and convergence in training-free guidance, which are presented in Tables 3 and 4 of the PDF, addressing Weakness 3 highlighted by Reviewer uPWj.


These changes will be integrated into the revised manuscript and we will carefully correct the typos in the revised manuscript. Please don't hesitate to let us know of any additional comments on the manuscript or the changes.

[A] Guidance with spherical gaussian constraint for conditional diffusion. arXiv:2402.03201.

[B] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method. arXiv:2312.12030.

[C] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.

Meta-review: The reviewers and I are in agreement that the paper presents a well-isolated study and contribution on using augmentations and Polyak step sizes for improving Training-free Loss-based Diffusion Guidance.
Decision: Accept (poster)