TFG: Unified Training-Free Guidance for Diffusion Models
Authors: Haotian Ye
Haowei Lin
Jiaqi Han
Minkai Xu
Sheng Liu
Yitao Liang
Jianzhu Ma
James Zou
Stefano Ermon
Venue: NeurIPS 2024 spotlight (2024)
Keywords: diffusion model
conditional generation
training-free guidance
Primary area: diffusion_based_models
TL;DR: We propose a training-free guidance framework that unifies existing algorithms and achieve SOTA performance in our proposed comprehensive benchmark.

Abstract: Given an unconditional diffusion model and a predictor for a target property of interest (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. Existing methods, though effective in various individual applications, often lack theoretical grounding and rigorous testing on extensive benchmarks. As a result, they could even fail on simple tasks, and applying them to a new problem becomes unavoidably difficult. This paper introduces a novel algorithmic framework encompassing existing methods as special cases, unifying the study of training-free guidance into the analysis of an algorithm-agnostic design space. Via theoretical and empirical investigation, we propose an efficient and effective hyper-parameter searching strategy that can be readily applied to any downstream task. We systematically benchmark across 7 diffusion models on 16 tasks with 40 targets, and improve performance by 8.5% on average. Our framework and benchmark offer a solid foundation for conditional generation in a training-free manner.

TFG: Unified Training-Free Guidance for Diffusion Models
Haotian Ye
Haowei Lin
Jiaqi Han'*
Minkai Xu' James Zou'
Sheng Liu'
Yitao Liang"
Jianzhu Ma
Stefano Ermon
Stanford University
Peeknng University
TTinghua University
Abstract
Given an unconditional diffusion model and a predictor for target property interest (e.g., a classifier), the goal training-free guidance to generate samples with desirable target properties without additional training. Existing methods, though effective in various individual applications, often lack theoretical grounding and rigorous testing extensive benchmarks. As result, they could even fail on simple tasks, and applying them to new problem becomes unavoidably difficult. This paper introduces novel algorithmic framework encompassing existing methods as special cases, unifying the study of training-free guidance into the analysis of an algorithm-agnostic design space. Via theoretical and empirical investigation, we propose an efficient and effective hyper-parameter searching strategy that be readily applied to any downstream task. We systematically benchmark across diffusion models on 16 tasks with 40 targets, and improve performance by 8.5% on average. Our framework and benchmark offer solid foundation for conditional generation in a training-free manner."
Introduction
Recent advancements in generative models, particularly diffusion models [61, 21, 62, 66). have demonstrated remarkable effectiveness across vision [65, 48, 52], small molecules [74, 73, 24], proteins[1, 72], audio [35, 29], 3D objects [40, 41], and many more. Diffusion models estimate the gradient log density (i.e., Stein score, [67]) of the data distribution [65] via denoising learning objectives, and can generate new samples via an iterative denoising process. With impressive scalability to billions of data [58], future diffusion models have the potential to serve as foundational generative models across a wide range of applications. Consequently, the problem conditional generation based on these models, i.e., tailoring outputs to satisfy user user-defined criteria such labels, attributes, energies, and temporll information, is becoming increasingly important [63, 21.
Conditional generation methods like classifier- based guidance (66, 7] and classifier-free guidance [23] typically require training specialized model for each conditioning signal (e.g.. coniiiicniiii classifier or text-conditional denoiser). This resource-intensive and time-consuming process greatly limits their applicability. In contrast, training-free guidance aims generate samples that align with certain targets specified through an off-the-shelf differentiable target predictor without involving any additional training. Here, target predictor can be any classifier, loss function, probability function, or energy function used to score the quality of the generated samples.
In classifier-bast based guidance [66, 7]. where noise- conditionl classifier is specifically trained to predict the target property on both clean and noisy samples, incorporating guidance in the diffusion
"qqual contribution. Corresponding to mailto: www.hhaayneetttaannaooooo edu. Code is available at hattps://githin com wwmm//oolleee//Taainiiii Free- Prreee-Gidaan
Free-Guidaeeee Guidence.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(b)
FreeDoM-1 FreetDoM-4
TFG UGD
80
LGD
MPGD
MPGD
Fole
70
Accuracy (%)
UGD-2
treining-
UGD-4
TFG-4
LGD
DPS
60-
40
Search space for TFG
30
40
120
160
(d)
Audio inpainting
Audio declipping
Combined guidance
Lost fraunes
(gender, hair color) (aan, blond)
impuint with TFG
decli with TFG
More tow high frequency
())
Style transfer
(g)
Fine-grained generation
pickup truck
canal Venice
white
Fine-grained spieces "fairy
"The Storry Night"
Quantum property guidance
image
Statue of Liberty Concentric in Minecruft
Elephant
Low polarizability
High polaricability
circle
Figure 1: (a) Illustration of the unified search space of our proposed TFG, where the height (color) stands for performance. Existing algorithms search along sub-manifolds, while TFG results in improved guidance thanks its extended search space. (b) The label accuracy (higher the better) and Fréchet inception distance (FID, lower the better) of different methods for the label guidance task on CIFARIO [30], averaged across ten labels. Ours (TFG-4) performs much closer training- based methods. (c~>h) TFG generated samples across various tasks in vision, audio, and geometry domains.
process is straightforward since the gradient of the classifier an unbiased driving term. Training- free guidance, however, is fundamentally more difficult. The primary challenge lies in leveraging target predictor trained solely on clean samples offer guidance on noisy samples. Although various approaches have been proposed [18, 63, 6, 2, 78] and are effective for some individual tasks, theoretical grounding and comprehensive benchmarks are still missing. Indeed, existing methods fail to produce satisfactory samples for label guidance even simple datasets such as CIFAR 10 (Figure 1). Morcover, the lack of quantitative comparisons between these methods makes difficult for practitioners to identify appropriate algorithm for new application scenario.
This paper proposes novel and general algorithmic framework for (and also named as) Training Free Guidance (TFG). We show that existing approaches are special cases of the TFG as they correspond particular hyper-parane papre-parreeter subspace our unified space. other words, TFG naturally simplifies and reduces the study of training- free guidance, well as the comparisons between existing methods, into the analysis hyper-parameter choices in our unified design space. Within our framework, we analyze the underlying theoretical motivation of each hyper-parameter and conduct comprehensive experiments to identify their influence. Our systematic study offers novel insights into the principles behind training-free free guidance, allowing for transparent and efficient survey of the problem.

---

Based on the framework, we propose a tasks. We comprehensively benchmark TFG and existing algorithms across 16 tasks (ranging from images to molecules) and 40 targets. TFG achieves superior performance across all datasets, outperforming existing methods by 8.5% on average. In particular, excels in generating user- required samples in various scenarios, regardless of the complexity of targets and datasets.
searching strategy for general downstream
In summary, we (I) propose TFG that unifies existing algorithms into design space, (2) theoretically and empirically analyze the space to propose an effective space-searching strategy for general problems, and (3) benchmark all methods on numerous qualitatively different tasks to present the superiority of TFG and the guideline for future research in training- frree conditional generation algorithms. This advancement demonstrates the efficacy of TFG and establishes a robust and comprehensive benchmark for future research in training- free conditional generation algorithms.
2 Background
Generative diffuusin model. A generative diffusion model is neural network that can be used to sample from an unconditional distribution (a( with the support on any continuous sample space
X' [21, 62, 64, 27]. For instance, X' could be [-1,1] 1jxxd3x representing the RGB colors of d x images [4, 22], or 3d representing the 3D coordinates of molecules with atoms [24, 74, 73). Given a data zo sampled from Po(=), a time step t E [T] & {I..... T. a corresponding noisy datapoint is constructed I; vā + vI - āyε where monotonically decreasing parameters used to control the noise level. Following [21], we further define a, = á for t > and a;2 0j. The diffusion model eg X' x [T] + X' parameterized by 0 is trained to predict the noise that was added on D with p.d.f pe = Jao (ojo 20p day.? In theory, this corresponds to learning the score of pr( [65], i.c.,
(O, I) and { ) is a set of pre-defined
arg min
Pao- -ell =-VI- ac V.V log
For sampling, start from Zy (0, I) and gradually sample -;-1 Pt (3t-1 |ax)- This conditional probability is not directly computable, and practice, DDIM [62] samples x;-1 via
Ct - v Vā
Ct Var + V āi- of
(2)
vI ae
where {a.)__ DDIM parameters, N(O, I), and
Poje m(x:)) A Dt. VI (re
Var
is the predicted sample given Fy. According to the Tweedie's formula [1], 20 equals to the conditional expectation B[ [x,] under perfect optimization of eq in Equation (1). It has been theo- retically established that the above sampling process results in By ~ p(x) under certain assumptions.
Target predictor. For a user required target c, we use a predictor fe(x) X' + R, U {0} 3 to represent how well sample æ is aligned with the target (higher the better). Here fe(x) can be conditional probability Po(c)z) for label [62, 14], Boltzmann distribution exp" (m) for any pre-defined energy function e. [31, 63, 38), the similarity of two features [47], or even their combinations. The goal is to samples from the conditional distribution
fc( 5& Po Po()
"In this paper. represent the probability at time step and the conditional probability æ at time step given at time step s. 'Here can has any mathematical form. We assume in this paper that fa(x) has finite two norm, i.c. X 1 (x:) < +00, such that the probabilistic explanation is well-defined.
p(æ) to represent the probability density function (p.d.f)), and pc(x).P Prja (ar (æ/()) to
3

---

Training-basee guidance for diffusion models. [66] proposes to train a time-dependent classifier to fit fe(a,, & (lm This can be regarded as predictor over noisy samples. Since
V mx Pogpc( Va. log / Prjo p l
Vx, log / pr (2:s) (co
m0
Va, log + V log fc
(5)
if we denote the trained classifier (xx;) (that implicitly depends e and model parameters),
we can replace E0 t) in Equation (3) by (x),,) /I ae Vx, log f(xe upon sampling to
obtain unbiased sample 2o Po (oollc)). On the other hand, [23] proposes the classifier-free diffusion guidance approach. Instead of training a time-dependent predictor f. it encodes conditions c directly into the diffusion model as c0 c, t) and trains this condition-awee aware diffusion model with sample- condition pairs. Both methods have been proven effective when training resources are available.
This paper in contrast focuses on conditional generation in a training-free manner: given diffusion model (0(2, i) and an off-the-shelf target predictor [(x) (we omit the subscript e below), we aim generate samples from Po (n(ccc) without any additional training. Unlike training- based methods that can accurately estimate f( f), training-fres guidance is significantly more difficult since involves guiding noisy data æ; using fxx defined over the clean data space.
2.1 Existing algorithms
Most existing methods take advantage of the predicted sample zoje defined in Equation (3) and use the gradient of f(x) for guidance. We review and summarize five existing approaches below, and provide a schematic and a copy of pseudo-code in Appendix B for the sake of reference. Due to the variety in underlying intuitions and implementations, coupled with lack of quantitative comparisons among these methods, it is challenging to discern which operations are crucial and which are superftuous, a problem we address in Section 3.
DPS [6] was initially proposed to solve general noisy inverse problems for image generation: for a given condition and a transformation operator A. we aim to generate image such that [lA(x))y)]
is small. For instance, in super-resolution task [71]. the operator A is a wwwn-sampling operator, and is a cew-rusolution image. DPS replaces V log f (z in Equation (5) by Va, ax log f (m(x))). As suggested in [63], this corresponds to a point estimation of the conditional density (ojo(Zo|
LGD [63] replaces the point estimation in DPS and proposes estimate f(x,, t) with a Gaussian (æ,,t), t), where the expectation is computed using Monte-Carlo sampling [59]. kernel Ex (0 frx
FreeDoM [78] generalizes DPS by introducing a 'recurrent strategy" (called "time-trave strat- egy" [39, egy"'39, 10, 70]) that iteratively denoises x;-1 from æ, and adds noise z;-1 to regenerate æ, back and forth. This strategy empirically enhances the strength of the guidance at the cost of additional computation. FreeDoM also points out the importance of altering guidance strength at different time steps 1, but a comprehensive study on which schedule is better is not provided.
MPGD [18] is proposed for manifold preserving tasks, e.g., the target predictor is supposed to generate samples on a given manifold. It computes the gradient of log f(wuj) to Toje instead of 2r, i.e., Vaay log (aoje) to avoid the pocpppropagttion through the diffusion model eg that is highly inefficient. This strategy is effective in manifold-preserving problems, but whether it can be generalized to general training-free problems unclear. In addition to the computation difference, theoretical understanding on the difference between gradients to Pojr and æ, is missing.*
UGD [2] builds on FreeDoM, with the difference that additionally solves backward optimization problem An = arg m f f(Zojt + A) and guides Zoj+ and H; simultaneously. UGD also implements the "recurrent strategy" further improve generation quality.
'MPGD additionally proposed to use an auto encoeer to improve the quality of Tojt However, auto- encoder is usually inaccessible or requires training; thus, don't apply it in our training freee scenario.

---

3 TFG: A Unified Framework for Training-free Guidance
Despite the array of algorithms available and their reported successes in various applications, we conduct a case study on CIFARIO [30] to illustrate the challenging nature of training-free guidance and the insufficiency existing methods. Specifically, for each of the ten labels, we use the pretrained diffusion model and classifiers from [7. 9] to generate 2048 samples, where the hyper- peeem--rsseters are selected via a grid search for the fairness of comparison. We compute the FID and the label accuracy evaluated by another classifier [20] and present results in Figure 1. Even in such relatively simple setting, all training-free approaches significantly underperform training- graining-based guidance, with significant portion of generated images being highly unnatural (when guidance is strong) or irrelevant to the label (when guidance is weak). These findings reveal the fundamental challenges and highlight the necessity of comprehensive study. Unfortunately, comparisons and analyses of existing approaches are missing or primarily qualitative, limiting deeper investigation in this field.
3.1 Unification and extension
This sections introduces our unified framework for training-free guidance (TFG, Algorithm 1) and formally defines its design space in Definition 3.1. We demonstrate the advantage of TFG by drawing connections between TFG and other algorithms to show that existing algorithms are encompassed as special cases. Based on this, all comparisons and studies of training-free algorithms automatically become the study within the soper-parammter space of our framework. This allows to analyze the techniques theoretically and empirically, and choose an appropriate rrper-parameter for a specific downstream task efficiently and effectively, as shown in Section 4.
Algorithm Training-FFrr Free Guidance
1: Input: Unconditional diffusion model ca, target predictor f' guidance strength p. 14,7, number of steps T, Nuwor, Niter
2: ær ~N(0,)
3: for t- wrt==T.--. -T....,, do
Define function [(x) Es. +5V1 a.8)
5: for r 1.... Neww do
6:
Zoit = (x: v1 1 azeo( t))/val
> Obtain the predicted data
7:
A. P: Va. log (
8:
Ao = Å. HeVan
V aoje log (wo 2 Ao)
b. Iterate Nuer times starting from A. = 0
9:
E = Sample v. Cojt. 1) d./Va, + var-1Ao
D Sample follows Equation (2)
10: w.
a
b Recurrent strategy
11: end for
12:
end for Output: Conditional sample æo
13:
Definition 3.1. Given denoising step T. the hyper-parameter space (design space) of Algorithm defined as
HTFG = {NNrccrr Nuer PsP, 14) Nresur, Nreour E N, 2 0 (R+ U (0)))²}
(6)
We use HYFG represent the complete hyper-parameter space and HTFG STTrG(NNcccr (Nrecur = No) represent the subspace constrained on Niesur No-
Definition 3.1 defines the
in HIFG is an instantiation of the framework. Intuitively, Nrecur controls the recurrence of the algorithm, Niter controls the iterating when computing Ao (Line 8), 7 controls the extent we smooth the original guidance function of (Line 4), and p. μ control the strength of two types of guidance (Lines and 8). comprehensive explanation of the effect of each paprr-parameter can be found Section 3.2.
space spanned by TFG, where one
Below is the major theorem showing that all algorithms presented in Section 2.1 correspond to special cases of TFG, thus unifying them into our framework and obviating the need for separate analyses.
Theorem 3.2. The hyper-parameter space of
MPGD [18] HMPGD is equivalent to HIFo = Nitr 1, p = 0,, = 0).
LGD [63] HLoD is equivalent to HyFo Nrewu 1, Nuer = 0, f 0).
5

---

UGD [2] HUGD is equivalent to HIFG( = 0).
DPS [6] Hops equivalent to Hrva Noour 1, Naie 0, 0,7 0).
FreeDoM [78] HrreeDoM is equivalent to (NFer = 0, μ = 0,5 = 0).
The complete analysis and proof of Theorem 3.2 is postponed to Appendix C. It implies that existing algorithms are limited in expressivity, covering only a subset of HTrG. In contrast, TFG covers the entire space and is guaranteed to perform better. In addition, TFG streamlines nuances between existing methods, allowing for a unified way to compare and study different techniques. Consequently, the versatile framework that TFG provides can simplify its adaptation to various applications.
3.2 Algorithm and design space analysis
We now present a concrete analysis of TFG and its design space H in detail. Similar to standard classifier-based guidance, TFG guides æ at each denoising step t. To provide appropriate and informative guidance, TFG essentially leverages four techniques for guidance: Mean Guidance (Line 8) controlled by Nites J., Variance Guidance (Line 7) controlled by p. Implicit Dynamic (Line 4) controlled by 5, and Recurrence (Line 5) controlled by Nrecur-
Mean Guidance computes the gradient of f(x) to Toje and is the most straightforward approach. However, this method can yield inaccurate guidance. To show this, notice that under perfect optimiza- tion we have Z = [2 azs], and when po (OEExo [o ær:]) is close to zero, the predictor has rarely been trained data from the region close Zoje. making the gradient unstable and noisy. To mitigate this, one can iteratively add gradients of f(x) to Zoje. encouraging Znje to escape low-probability regions.
Variance Guidance provides an alternative approach for improving the gradient estimation. The reason why we dub it variance guidance might be ambiguous, as the only difference is that the gradient is taken with respect to I; (Line 7) instead of Zoje (Line 8). The lemma below demonstrates that this essentially corresponds to covariance re- scaled guidance.
Lemma 3.3. If the model is optimized perfectly, i.e., c2(2,)) -V1 āi V log Pc(),, we have
Vot Σofe V M0oee f(
A
1 at
where Σoj= A Ie (o)e(2x (z 2( T dæ is the covariance of zo (2
Lemma 3.3 suggests that variance guidance refines mean guidance by incorporating the second- order information of x 1o, specifically considering the correlation among components within Toje- Consequently, positively correlated components could have guidance mutually reinforced, while negatively correlated components could have guidance canceled. This also implies that mean guidance and variance guidance are intrinsically leveraging different orders of information for guidance. In TFG, variance guidance is controlled by Pa.
Implicit Dynamic transforms the predictor into its con- volution via a Gaussian kernel .V (0, 7(1 I). This operation is initially introduced by LGD [63] to estimate Paj 22). However, it is unclear why the form is pre- selected as Gaussian distribution. We argue that this tech- nique is effective because it creates an implicit dynamic on oojz- Specifically, starting from Poje: it iteratively adds noise to Zo 1. evaluates gradient, and moves Toje based on the gradient. The repeating process converges to the density proportional to f(x) when Niter goes to infinity, driving & to high-density regions. This explanation is justified by Table 1: the performance remains nearly un- changed as we gradually decrease the number of Monte-Carlo samples in estimating the expectation (Line 4) down to 1, implying that the preciseness of estimation is not essential, but adding noises is.
Table 1: Influence of the number of Monte-Carlo samples in estimating the expectation of Line 4. Both the FID and the accuracy remain unchanged when #Samples varies, suggesting that the
number of samples is less important. More details are in Appendix E.I.
Variance only
Mean only
# #Samples
FID Acc(%)
FID
Acc(S))
90.6
65.8
101
36.2
2
91.0
65.2
100
35.6
4
90.7
64.9
99.7
36.2
Recurrence helps strengthen the guidance by iterating the previous three techniques to obtain # and resample a; back and forth. This can be understood as an - 1 where Line 6~9 corresponds to the drift term and 1 -> 2,, (Line 10) the white noise term. Intuitively, it finds a trade-off between the error inherited from previous steps (the more you recur, the
process[42] on

---

InageNe
340
2.4
-24
200
2.50
Figure 2: Comparison of three structures in Equation (8) of p and μ on CIFARIO and ImageNet, under different choices of the rest of μ. and similarly for p. Results are averaged across all labels. The comparative relationship between structures remains unchanged when the rest of the parameters vary.
in HIGG We set p 0, 0 when studying structures
less previous error stays) and the accumulated error in this step (the more you recur. the more error in the current guidance you suffer). Empirically, we also find that the generation quality improves and then deteriorates as we increase Nrecur-
4 Design Space of TFG: Analysis and Searching Strategy
Admittedly, a more extensive design space only yields a better performance if an effective and robust hyper-parameter searching strategy can be applied. For example, arbitrarily complex neural networks are guaranteed to have better optimal performance than simple linear models, but finding the correct model parameters is significantly more difficult. This section dives into this core problem by comprchensively analyzing the hyper-parameter parameter space structure of HTFG. and further proposing a general searching algorithm applicable for any general downstream tasks.
The hyper- syper-parameters of HIKG can be categorized into two parts: and indepenneenn sacalars Nucurs Nites While a grid search potentially result in the best performance, performing such an extensive search in HTFG is highly impractical, especially considering the vector parameters P; J.. Fortunately, below we demonstrate that, if we decompose into p (4) (same for 14) where is a scalar and (l) is a "structure" (a non-negative function) consistently better than others regardless of the
time-dependent vectors j.,
such that Σ. (1) T, then some structures other hyper-parameters. This allows to poc-locate an appropriate structure for the given task and efficiently optimize the rest of the scalar hyper- parameeaasssss. Our analysis is conducted on the label guidance task on CIFAR-I0 [30] and ImageNet [55], with experimental settings identical to Section 3,
Structure analysis. Motivated by the default structure selected in UGD and LGD, we consider three structures for both sp (t) and s,,(t (t) as
(increase), s(t) (1 ar) ET aa))
(t))
(decrease), s(t) 1(constant..
1t=1 ag
These structures are selected be qualitatively different, while each is justified to be reasonable under certain conditions [18, 78, 2]. We leave the study of more structures to future works. The rest of the parameters are grid-searched for the comprehensiveness of the analysis. For (t), we set Neeuu {1,2, and p {0.25, 0.5, 1.0, 2.0, 4.0); and for s, (t), we set Nuer {1, 2, 4] and á (0.25, 0.5, 1.0, 2.0, 4.0). We run label guidance for each configuration and each of the ten labels on CIFAR10 (four labels on ImageNet, due to computation constraints).
As presented in Figure the relationship between different structures remains unchanged when the rest of the parameters vary. For instance, on both datasets, the Validity-FID performance curves consistently move top-left (implying a better performance) when switch from "decrease" structure (red lines) to "constant" structure (yellow lines) to "increase" structure (blue lines) for both l and different values of Nessur and Niter- This invariant relationship is essential as it allows for an efficient hyper-parameters search in HTFG by first determining appropriate structures for s (f), sp (1) under a simple subspace, and then selecting the rest scalar parameters.
7

---

Table 2: List of task types we benchmark. Each task is run with multiple individual targets (38 in total). We evaluate the guidance validity (how well sample is aligned with the target predictor) and the guidance fidelity (how well a sample is aligned with the unconditional distribution) according to the task type.
Diffusion Model
Task-ID
Targets
Guidance Validity Guidance Fidelity
Gaussian deblur
LPIPS L
FID L
Cat-DDPM
Super-resolution
LPIPS +
FID 1
Combincd guidanec (gender+age)
genders 2 ages
Accuracy (%) f
KID (log)
CelcbA-DDPM
Combined guidance (gender+hair) 2 genders hair colors
Accuracy (%) f
KID (log)
CIFARIO-DDPM
Lahel guidance (CIFARIO)
10 lahels (0, ..., 9)
Accuracy (%) f
FID 4
Label guidance (ImageNet)
4 lahels (111, 444)
Accuracy (%) f
FID +
ImageNet-DDPM
Fine-grainee grained guidance
4 labels (111. 444)
Accuracy (%) f
FID
Stable-Diffusion
Style transfer
styles
Style score
CLIP score f
MoDM
Quantum Properties (x6)
Property distribution
MAE
Valid ratio
Audio declipping
DTW (%)
FAD $
Audio-Diffusion
Audio inpainting
DTW (%) L
FAD
Computation cost analysis. Among the scalars parameters, .Vreur and Nieer directly influence the total computational cost, while p. F. 7 do not. With a certain range. performance increases when the value of Nuceur, Nuer increase", and the trade- off between generation quality and computation time is presented in Figure 3: recurrence leads to a Neecur times cost with clear performance gain; iteration (on To 2) results in less increase of com- putation time, and its effect plateaus. In practice, users can determine their values based on compu- tation resources, but an upper bound of suffices to unlock a near-optimal performance.
Mhc 4
135-
3.25
Figure 3: Accuracy and FID on CIFARIO under different Niceur and Niter sp(t), st() are fixed to "increase" structure, and p = 7 = O.
Searching strategy. The above analysis successfully simplify the task task- specific search problem without significant performance sacrifice. It remains to be decided the scalar values p. Jú, 5. Here we propose strategy based on beam search to effectively and efficiently select their values. Specifically, our searching strategy starts with an initial set T = [[Piiit Mumit Timi)) J. where these initial values are small enough to approximate TFG as an unconditional generation. At each searching step, for each tuple in T', we separately double the values of p. p. and 5 to generate up to 3|T'| new configurations. We conduct a small-sized generation trial for each new configuration and update T' to be the top K configurations with the highest evaluation results that are determined by user requirements (e.g., accuracy, FID, or a combination). This iterative process is repeated until T stabilizes or the maximum number of search steps is reached. Notice that this process is conducted with a much smaller sample size, and consequently, the computation time is highly controllable.
5
Benchmarking
This section comprehensively benchmarks vriniin-free guidance under the TFG framework and the design searching strategy in Section 4. We consider datasets, 16 different tasks, and 40 individual targets with a total experimental cost of more than 2,000 A100 GPU hours. For comparison, we also run experiments for each of the existing methods (where the design searching is conducted in the corresponding subspace). All methods, tasks, search strategies, and evaluations are unified in our codebase, with details specified in Appendices D and E.
5.1 Settings
Diffusion models. (1) CIEARIO-DDPM [48] is a U-Net [54] model trained on CIFARIO [30] images. (2) ImageNet-DDDM [7] is an larger U-Net model trained on ImageNet- Ik [55] images. (3)
Cat-DDPM is trained on Cat [12] images. (4) CelebA-DDPM is trained on CelebA-HQ dataset [26]
that consists millions of human facial images. (5) Molecule-EDM [24] is an equivariant diffusion
We notice that the FID worsens when Nrecur Nuier are too large (e.g.. 10).
8

---

Table 3: Benchmarking TFG and existing algorithms on 16 task types and individual targets. Each cell presents the guidance validity'genrratinn fidelity averaged across multiple targets in the task (e.g. labels, image styles). The best guidance validity is bold, and the second best underline. The relative improvement of guidance validity is computed between TFG and the existing method with the highest guidance validity.
TDsk-ID
DPS
LGD
FreeDoM
MPGD
UGD
TFG
Rel. Improvement
Deblur (1. J)
0,390
0.270/ 85.1 0.245/ 87,4 0.177 / 69.3 0.200/ / 69.3 0.150/ 64.5
+15.3%
Super resolution (L.
0.420/ 109
0,360 / 96.7
0.191 74,5 01283/ 82.0 0.249 / 75.9 0.190/ 65.9
+0.524%
Gender+Age (1:4)
71.6/-.26
52.0/-5.10
(8.7(-3.8)
68.6/ -4.99
25.1/-4.37
75.2/-3.86
+0.133%
Gender-Hair (1:4)
73.0/-3.90
55.0/-5.00
6-.1
63.9/ -433
71.3/-4.12
76.0/
+4.11%
CIFARIO (t:4)
50.1 172
32.2/ 102
34.8/ 135
38.0 88.3
/9.9
52.0/91.7
+3.59%
ImageNet (t-4)
38.8 / 193
11,5/210
19.7 / 200
6.80 239
25.5
40.9/ 176
+5.41%
Fine-grained (t.4)
0.00/344
0.48 246
0.58 258
0.58 244
1.07
/55
/
+18.7%
Style Transfer (u,
5.06//117
5.26/ 31.3
31.2
4/08 / 31.5
4.97
31.5
3.16/29.0
+22.5%
Polarizability <r (4.
51169.7 92.3
7.155 / 84.3 5.922 88.0
4.26 / 88.4
/
3.90 / 84.2
+8.45%
Dipole μ (J.1)
77.3/
/
1.35 / 89.5
1.51 73.5
1,56 57.6
1.33 / 74.9
+1.48%
Heat capacity Co (4- t)
5,26/
3.77
2.84/ 90.9
2.86 / 86.1
3.02 84.0
2.77 / 85.5
+2.57%
Highest MO energy 41000 (L. t>
0,744 83,8
0.664 / 66.4
0.623 62.3
0.554/ 53.4
0.582 / 58.2
0.568 77.3
-2.53%
Lowest MO energy FLUMO 1)
NA/
/
1,16/ 90.2
/ 82.2
1.27 /85.1
0.984/ 80.
+7.17%
MO energy gap (4: 1)
1,38 75.7
1.17 /5.3
88.5
1.2 / 72.5
75.7
0.893/
+16.7%
Audio declipping (J. L)
633 3.66
/
126 126/0.173
178 / 0.442
150 0.262
1.170
+19.8%
Audio inpainting (1- ()
643/4.71
1
41.3 / 0.08
608 / 4.63
116/0..3
/.03
+12.1%
model pretrained on molecule dataset QM9 [50] that performs molecule generation from scratch. (6) Glable-Diffusion (v7.5) [53] is a latent text-to-image model that generate images with text prompts. (7)
is audio diffusion model based on DDPM trained to generate mel spectrograms of 256x256 corresponding to seconds of audio.
Tasks. Our tasks (Table 2) cover wide range of interests, including Gaussian deblur, super-resolution, label guidance, style transfer, molecule property guidance, audio declipping, audio impainting, and guidance combination. Each task is run on multiple datasets or with multiple targets (e.g., different labels, molecular properties, styles).
Other settings. We consistently set the time step T 100 and the DDIM parameter n= 1. We consider Niecur 1, Niner - 4 and use single sample for Implicit Dynamic (Line 4) throughout all experiments and methods for fair comparison. For TFG, the structures of p and μe are set to "increase" and the scalars p. jú, 7 are determined via our searching strategy. We follow the setting in original papers if they specify their hyper-parameters. blueFor specific tricks in the code that are not mentioned in papers, we choose to align with original papers. Otherwise, values are determined via searching with 1/8 of the sample size and a maximum search step of 6. For fairness of comparison, we use accuracy as the metric during the search and compare different algorithms on the metric, but we report both accuracy and FID.
5.2 Benchmarking results
We compare all six methods in Table 3. TFG outperforms existing algorithms in 13 over 14 settings, achieving an average guidance validity improvement of 7.4% compared to the best existing algorithm. Notice that do not compare with the best algorithm in terms of generation fidelity because obtaining high realness samples is not our objective in training-free guidance, and an unconditional model suffices generate high realness samples (with extremely low validity). Interestingly, different methods achieve the second best performance on different tasks, suggesting the variance of these methods, while TFG is consistent thanks to the unification.
We want to highlight that despite the superior performance of TFG, the key intention of our ex- periments is not restrained to comparing TFG with existing methods, but more importantly to systematically benchmark under the training-free guidance setting to see how much we have achieved in various tasks with different difficulties. Below go through each task separately and conduct relevant ablation studies to provide a more fine-grained analysis.
sine-grained label guidance. In addition to the standard label guidance, we for the first time study the out-rf-disttibutton fine-gra fine-grained label guidance under the training-free setting, a problem where no existing training-based based methods are available. We consider the bird-species guidance using an
"attps://huggiingaaco.o http://gww..g.gttte//////////eee wtt.:iiiooouuiiooiifuiiioooo.oo.

---

EfficienNet trained to classify 525 fine- -neeggraidd bird species. This problem remains highly difficult for leading text-to-image generative models such as DALLE. Under recurrence, TFG can generate at most 2.24% of accurate birds, compared with the unconditional generation rate of O.
Recurrence on label guidance. We go back the failure case we study in Section 3, i.e., the stan- dard label guidance problem on CI where the training- method offers an 85% accuracy, while the accuracy of TFG without recurrence accuracy 52% only. As presented in Table 4, increasing Nieour significantly closes the gap from 33% to 8%. Similar improvement is observed in other datasets as well.
Table 4:
The accuracy FID for TFG with different recurrence step Nrecur three label guidance datasets, averaged across all labels.
Recurrence
CIFARIO
$2.0/91.7
66.8 88.7
77.1
ImageNet
40.9/ 177
52.3 / 163
59.8 / 165 2.24/259
Fine-grained
1.27 /
1.66/ 259
Multiple guidance and bias mitigation. We next consider the scenario with multiple targets: control the generation of human faces based on gender and hair color (or age) using two predictors. It is well known that the label imbalance in CelebA-HQ causes classifiers to focus on spurious correlations (76), such as using hair colors to classify gender, a biased feature we aim to avoid. The stratified performance of TFG on "gender + age" and "gender + hair" guidance are presented in Table 5. Despite the highly disparate performance, training-free guidance largely alleviates the imbalance: only % of images in CelebA are "male + blonde hair", while the generated accuracy is 46.7%.
Table 5: The accuracy of multi-label guid- ance on CelebA, where labels 0 and I cor- respond to female and male (gender), non- blonde and blonde (hair color), and young and old (age). The accuracy is lower for mi- nority groups, indicating an implicit bias in the generation process. Despite this, it is still much higher than unconditional generation.
Target label
0+0
0+1
1+0
1+1
gender + hair
92.2
72.7
89.8
46.7
gender + age 92.9
73.6
93.6
69.1
Molecule property guidance. To our knowledge, we are the first to study training-free guidance for molecule generation. We interestingly find in Table that TFG is effective in guiding molecules towards desirable properties, yielding the highest guidance validity on 5 out of 6 targets with 5.64% MAE improvement over existing methods, verifying the generality of our approach as unified framework in completely unseen domains. Notice that, unlike images, molecules with better validity usually have lower generation fidelity, a finding reflected in previous work [3].
Audio Guidance. We extend our investigation to the audio modality, where TFG achieves significant relative improvements over existing methods. Given that the audio domain is rarely explored in training-free guidance literature, our benchmarks will contribute to future research in this area.
Discussions and Limitations
Recently, training-free guidance for diffusion models has gained increasing attention and has been adopted in various applications. TFG is based on an extensive literature review over ten algorithmic papers for different purposes, including images, audio, molecules, and motions [34, 8, 43, 32, 19, 13, 16, 15, 45, 38, 68). While we incorporate several key algorithms into our framework, we acknowledge that encompassing all approaches is impossible, as it would make the unification bloated and less practical. We seek to find balance point by unifying most representative algorithms while keeping the techniques clear and easily studied.
An often discussed problem is why we care about training-free guidance, given the ever-growing community of language- generative models such as the image generator of GPT4, In practice, there are countless conditional generation tasks where the conditions are hard to accurately convey to or represent by language encoders. For instance, it can fail to under a complex property of molecule or generate CelebA-style faces. We give an illustrative analysis in Appendix A. Despite that training- ffree guidance is important, this paper does not systematically analyze what types of conditional generation are, in general, more suitable for the framework and what types are for language- based models. That said, training-fres free guidance is fundamentally difficult due to the misalignment between the training objective of target predictors and the diffusion, with a more detailed discussion in Appendix A.2. This paper does not comprehensively analyze this misalignment, and the gap between training-basee and TFG remains high in some tasks like molecule property guidance. We hope that future works can analytically dive into these problems.
10

---

References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneherger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 843-852, 2023.
[3] Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations, 2023.
[4] Georgios Batzolis, Jan Stanczuk, Pioola-Bibiane Schönlicb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111. 2021.
[5] Mikolaj Binikowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.0140) 2018.
[6] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80:102479 2022.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-8794, 2021.
[8] Kieran Didi, Francisco Vargas, Simon V Mathis, Vincent Dutordoir, Emile Mathieu, Urszula Komorowska, and Pietro Lio. A framework for conditional diffusion modelling with applications in motif scaffolding for protein design, 2024.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
[10] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstenn Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International conference on machine learning, pages 8489-8510. PMLR, 2023.
[11] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):1002-6614 2011.
[12] Jeremy Elson, John (JD) Douceur, Jon Howell, and Jared Saul. Asirra: A captcha that exploits anterest-aligned manual image categorization. In Proceedings of 14th ACM Conference Computer and Communications Security (CCS). Association for Computing Machinery, Inc., October 2007.
[13] Daniel Geng and Andrew Owens. Motion guidance: Diffusion-based image editing with differentiable motion estimators. arXiv preprint 2.Xv:22001./8885 2024.
[14] Alexandros Graikos, Srikar Yellapragada, and Dimitris Samaras. Conditional generation from unconditional diffusion models using denoiser representations. arXiv preprint arXiv:2306.01900 2023.
[15] Jiatao Gu. Qingzhe Gao, Shuangfei Zhai, Baoquan Chen, Lingjie Liu, and Josh Susskind. Control3diff: Learning controllable 3d diffusion models from single-view images. In 2024 International Conference on 3D Vision (3DV), pages 685-696. IEFF, 2024.
[16] Xu Han, Caihua Shan, Yifei Shen, Can Xu, Han Yang. Xiang Li, and Dongsheng Li. Training- free multi-objective diffusion model for 3d molecule generation. In The Twelfth International Conference Learning Representations, 2023.

---

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Yutong He, Naoki Murata, Chieh- Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Rsiing Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Representations, 2024.
[19] Carlos Hernandez- Olivan, Koichi Saito, Naoki Murata, Chieh-Hsie Lai, Marco A Martínez- Ramirez, Wei-Hsiang Liao, and Yuki Mitsufuji. Vrdmg: Vocal restoration via diffusion posterior sampling with multiple guidance. In ICASSP 2024-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 996-600. IEEE, 2024.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prohabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[22] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint M.Xi::2277.12598, 2022.
[24] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusson for molecule generation in 3d. In International conference machine learning, pages 8867-8887. PMLR, 2022.
[25] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Mision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 694-711. Springer, 2016.
[26] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint
[27] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23666 2022.
[28] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fri 'echet audio dis- tance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 2018.
[29] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwaave A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0977 2020.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial energy- based learning. Predicting structured data, 1(0), 2006.
[32] Jean-Marie Lemercier, Julius Richter, Simon Welker, Eloi Moliner, Vesa Välimäki, and Timo Gerkmann. Diffusion models for audio restoration. arXiv preprint arXhv:2402.09881,, 2024.
[33] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE conference on computer vision and pattern recognition, pages 1964-1971. IEEE, 2009.
[34] Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Con- trollable music production with diffusion models and guidance gradients. arXiv preprint arXiv:2311.0011: 2023.
12

---

[35] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-ane -ouuiio generation with latent diffusion models. arXiv preprint arXiv:2301. /2503, 2023.
[36] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022.
[37] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[38] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu, Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825-22855. PMLR, 2023.
[39] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pages 11461-11471, June 2022.
[40] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2844 2021.
[41] Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion-refinement paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.08530, 2021.
[42] Ross A Maller, Gernot Müller, and Alex Szimayer. Ornstein-uhlenbeee processes and exten- sions. Handbook of financial time series, pages 421-437, 2009.
[43] Eloi Moliner, Jaakko Lehtinen, and Vesa Välimäki. Solving audio inverse problems with a diffusion model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[44] Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69 84, 2007.
[45] Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang. Toshiaki Koike- Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug-and play conditional image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20850-20860, 2023.
[46] Kamal Nasrollahi and Thomas Moeslund. Super-resolution a comprehensive surveyy Machine vision and applications, 1211222-168,, 2014.
[47] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[48] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[49] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[50] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Lilienfeld. Quantum chemistry structures and properties 134 kilo molecules. Scientific Data, 1, 2014.
[51] Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388-394. Springer, 1992.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
13

---

[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (0684-10095,,)) June 2022.
[54] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted interventionnMIICAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. pages 234-241. Springer, 2015.
[55] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-25) 2521 2015.
[56] Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.0053 2015.
[57] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. arXiv preprint 1xXi::2102.08844,, 2021.
[58] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 252944, 2022.
[59] Alexander Shapiro. Monte carlo sampling methods. Handbooks in operations research and management science, 10:353-425,
2003. 2003.
[60] Herbert A Simon. Spurious correlation: A causal interpretation. Journal of the American statistical Association, (9(267):467-479 1954.
[61] Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages
PMLR, 2015.
[62] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:20000.02500, 2020.
[63] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference Machine Learning, pages 32483-32498 PMLR, 2023.
[64] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances neural information processing systems, 34:1415- 1428, 2021.
[65] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[66] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 I. 13456, 2020.
[67] Charles Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. In Proceedings of the Sixth Berkeley Symposium on Mathmatical Statistics and Probability, Volume 2: Probability Theory, volume 6, pages 583-603. University of California Press, 1972.
[68] Haoyuan Sun, Bo Xia, Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigm of text-to-image generation with preferences through f-divergence minimization. arXiv preprint 209007440.009774, 2024.
14

---

[69] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention, 2021.
[70] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023.
[71] Zhihao Wang. Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. IEEE transactions pattern analysis and machine intelligence, 43(10))3365-3333 2020.
[72] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe. Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas Milles, al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976)
2023.
[73] Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure I Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pages 38592-38610. PMLR, 2023.
[74] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang, Geodiff: A geo- metric diffusion model for molecular conformation generation. 2022.
gennraattiognnatitioon arXiv preprint drXiv:2203.02923,
[75] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized distr-idistiittionn detection. 2022.
[76] Haotian Ye, James Zou, and Linjun Zhang. Freeze then train: Towards provable representation learning under spurious correlations and feature noise. International Conference on Artificial Intelligence and Statistics, pages 8968 19-9900.. PMLR, 2023.
[77] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint 17X7v92206.1078,, 2(3):5, 2022.
[78] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[79] Ning Zhang. Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for fine- grained category detection. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 834-849 Springer, 2014.
[80] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
15

---

A The Motivation of Studying Training-frre Guidance
In this section, we argue that training-free guidance using off-the-shelf models a crucial and timely research problem deserving more atfention and effort. We begin by providing illustrative analysis that highlights the limitations of current strong text-to-image generative models, underscoring the necessity for training- -ree guidance. Furthermore, we assert that training- frree guidance remains significant challenge, with previous literature underestimating its complexity. Given its necessity and inherent difficulties, we call for increased focus from the research community on this problem and offer our benchmarks and code base to help accelerate progress in this area.
A.1 Failure case of image generation with GPT4
Hello ChatGPT, can you create figura of molecule with Polarizability =?
Sure! However, polarizability is property of molecule that indicates how easily its electron cloud can be distorted by external electric field. The numeric value of "3" you've mentioned needs a unit and more context to understand which molecule or lype of molecule it refers to. Commonly, polarizability is expressed units like A (cubic Angetroms) or other volume units.
Could you specify which malecule you are referring to, or provide additional details about its structure or composition? This would help accurstely generating representation the molecule.
Any visualization.
atoms
considering reasonable polarizability figure hydregen atems, forming a ring structure, exhibiss
malecule
properties.
I'll generate standard visualization the you. Let's
Hare 3D melecular madel of benzene, which features hexagonal ring structure. This visualization shows the arrangement of carbon and hydrogen atoms, with the typical aromatic double band alternation. You can view the image above.
Message ChatGPT
Figure 4: Prompting GPT4 to generate property guided molecules. It is hard for the image generator to understand the target and generate faithful samples. In this dialog, GPT4 claims to generate a benzene molecule but the sample is apparently not benzene. There are also many invalid carbon atoms with more than bonds and the polarizability target is not achieved.
It's hard for GPT4 to understand targets. In Figure 4, we ask GPT-4 to generate a molecule with polarizability a = 3, which is task we use to evaluate training -free guidance (refer to Figure 16 for visualization). We found that the GPT-4 generated molecule apparently invalid and unrealistic: the generated molecule contains many carbon atoms with more than 4 bonds (the maximum allowed number 4); and the generated molecule apparently not benzene which claimed by the text outputs. From this case we may understand that it is hard to follow diverse user-defined instructions for the foundational generative models, where the user-defined targets may subtle, fine- grained, combinatorial, and open-ended.
16

---

To this end, training- thining-free guidance offers two key advantages: (1) It allows for greater specificity in target requirements by enabling the use of differentiable objective function, making the generation process more steerable; (2) The objective function is plug-and-play facilitating the addition of new targets and tasks to pre-trained generative model. Since there no need retrain the diffusion or prediction models, this approach makes the generative process lightweight and applicable to various downstream tasks.
It's hard for GPT4 capture the targeted distribution. Another important metrit for training- free guidance is the flexibility of choosing diffusion models. For the same target, we can switch from different diffusion models to change the unconditional background distribution. For example, is hard for GPT4 to generate CelebA-like samples though it "knows CelebA dataset very well";
ChatGPT 4
Hello ChatGPT, do you know about CelebA dataset?
Yes,
(CelebFaces
Dataset) a large-
scale face attributes dataset designed research field of facial attribute recognition, face detection, and landmark (or facial part) localization. includes over 200,000 images celebrities, each annotated with 40 attribute labels (like "Smiling" "Young", "Wearing Glasses", etc.) and 6 landmark locations (eyes, nose, mouth corners). The dataset has been widely used in machine learning and computer vision research, particularly for tasks involving facial recognition and modification.
identify manipulate these attributes unseen images, advancing the capabilitiss technology. The dataset publicly available for non-commercial research purposes, making it popular choice for academic studies and experiments image processing and machine learning.
Researchers CelebA develop
algorithms
recognition
Can you generate an Image of face of young man in celebA style?
Here the generated image young the CelebA dataset,
@
Message ChatGPT
Figure 5: Prompting GPT4 to generate CelebA-like images. We first prompt ChatGPT to probe its knowledge of CelebA dataset and then ask it to generate young man figure in CelebA style. However, the generated figure is apprently not in the distribution of CelebA (refer Figure 12) for comparison.
The flexibility to use different diffusion models provides an opportunity to generate wider range of user-defined targets. With training-free guidance, individuals can select their preferred diffusion model establish the background distribution and use the prediction model to steer the generation towards specific properties. This approach may represent future direction for human-Al interaction.
17

---

DPS
DeSiner
LUp
Lquueer
0.8
MFGD
MPGD
UGD-1
UGD
uCD2
UG2
0.7
FreeDite
Fresbutherrio
baring
Validity
0.6
bradd (FID: 25)
MPGD-fake (FID: 28)
MPGD (FID: 53)
0.5
Training based
1-9
1-15
1-10
1-35
1-30
1-25
1-20
1-15
1-10
1-5
1-0
0.4
MPGD (lake)
49
1-45
1-40
1-35
1-30
1-25
1-20
1-15
1-10
F-
1-0
0.3
40
60
80
100
120
140
160
MPGD
FID
40
3:
1030
25
15
1010
F-5
F-0
Figure 6: (Left) The accuracy and FID of different methods under different settings on CIFARIO [30], average across ten labels and 2048 samples per label. The suffix number in UGD and FreeDoM represents recurrent step Nrwar and (fake) stands for synthetic setting where we apply a training- based classifier but set 0 and the same training-free guidance methods. A huge performance gap between different settings suggests the intrinsic difficulty of training-free -fren guidance. (Right) Illustration of generated "ship" using MPGD under different settings (top) and the sampling trajectory of the predicted clean image Zoje (down).
A.2
The fundamental challenge of training-free guidance
Despite the array of algorithms available and their reported successes in various applications, we conduct case study on CIFARIO [30] to illustrate the challenging nature of training-free guidance and the insufficiency of existing methods. Specifically, we compare the training- based approach and training-free approach for the label guidance task CIFARII 10, with the diffusion model pretrained by [7], the training-bann -based time-dependent classifier ((x 1) by [7], and the training-frre standard label classifier f(x) pretrained only on clean CIFARIO by [9]. and a "fake" training-free classifier defined as f(x 1) 100- The first serves as the oracle benchmark, while the second corresponds to the standard training-free free guidance. The third setting, as considered in LGD [63]. uses "fake" training-free free classifier since its parameters are shared across different time steps & during training, resulting in an implicit regularization that is not available for practical predictors. This setting serves as a comparison to help identify the difficulty of training-fres guidance.
Quantitative and qualitative results are shown in Figure 6. All training- free approaches significantly under-perform training-bass based guidance, with a significant portion of generated images being highly unnatural (when guidance is strong) irrelevant to the label (when guidance is weak). A more clear illustration on this can be found in Figure In terms of "fake" classifiers, it leads to a remarkable difference from real training-free classifiers even under identical experimental settings. It generates
1-m
Training-based (FID=44.8, Acc-826) FreeDoM-4 (FID-74.3, Acc-
TFG-4 (FIDD66.5, Acc-88%)
Figure 7: Illustration on CIFARIO dogs generated with different algorithms. Compared with training- based method, training-free free methods fall behind but TFG significantly outperforms existing methods.
18

---

much less messy images due to the implicit regularization from the training process, where noisy images are also "seen" (although t is fixed to upon guidance). Unfortunately, such types of predictors are inaccessible in practice (otherwise, we can use classifier- guidance directly). From the comparison, we know that the key challenge of training-free guidance the lack of "smoothing" classifier that can produce faithful guidance in the "unseen" noisy image space.
The observation largely uncovers the essential difficulties of training-free guidance, and motivates us to systematically study techniques that can improve generation quality. Unfortunately, comparisons between existing techniques are ambiguous since different methods are tested on distinct and primarily qualitative applications, which in turn hinders the in in-depth study in this field. To this end, we resolve revisit this complicated scenario and design a clear and comprehensive framework for training-free free guidance.
19

---

B Pseudo-code and schematics
We have presented the pseudo-code of TFG in Algorithm 1. Below, we provide copy the DPS (Algorithm 2). MPGD (Algorithm 3), FreeDoM (Algorithm 4), UGD (Algorithm 5), and LGD (Algorithm 6). Notice that LGD does not provide a pseudo-code, and we present their algorithm following their paper modification of DPS. We do not change the original algorithms' notations for reference. Please see the proof in Appendix C for the equivalence analysis. provide schematic of existing algorithms in Figure 8.
Algorithm 2 DPS Gaussian
Require: N. y. (G.)mm.. (6.) IN (0, I)
for N
do
+ so (z; i)
co - vo, (I; + (1 (4)
(0, 1)
#/-1 f- (0,-1- 23 + 1-
f
io + o;z
1-n;
2:-1 - 1 4. V 11y A(Ea))
end for return
Algorithm 3 MPGD for pixel diffusion models
1: ar T)
2: for f. T....., do
61 ~10,0
4: 20je -., - - VI aics (xt,, ())
requires manifold projection then
6: Zo = (M L (Zroes a))
7:
else
8:
Fo Zoie ct V suje 1 ((
9:
end if
10:
v /au-iooi
11: +V1 - ā,- of c 1) t.c
12:
end for
return 2o
Algorithm 4 FreeDoM + Efficient Time- Travel Strategy
Require: condition c, unconditional score estimator s(, (), time inde-indeeendent distance measuring funetion (c, ). pre- purameters B., ae learning Ps. and the repeat times of time travel of each step (F1...) .r].
Xr ~ [10,1)
T....: do
for i Pix.... do
61 (O I) clse O.
(1
x)) Beo(x,, () VBLes
Koi:
Jag (xt (1 G.)s(x., ())
8: (x..))
PIg
it's then
<g ((O.
x;
Pix: 1 V83
end
end for return
20

---

Algorithm 5 Universal Guidance (its a is our @)
Parameter:
Recurrent steps k, gradient steps m for backward guidance and guidance strength s(t,,
sampled from (O, 4), diffusion model cs. noise scales {av, Je=1, guidance function f. loss function L, and prompt
Required:
for 7.T 1, ... do
,..... do
Calculate 2p (r)oo)o)
Vos
Eo (=(..)) =00 1) s(1) V=,f(e,f(ão))
if > 0 then
Calculate A by minimizing {(c, f(ão + A)). with mn steps of gradient descent
Perform backward universal guidance by êg t- ês v /as/( (1 - as) Azo
end if
2t-1 <- -=,zz,, è ~N(1
Var/ar-122 + V1 a1/an-1c'
end for end for
Algorithm 6 LGD (from DPS)
Require: N, y. ((:)),,
for)
IN
(00 I)
for N
do
- so (T;, i)
2o - vor (r, + (1 o.)8)
~ (0,
21-1 (1-ج)_ 23 + 1 Do + 0,2
1-a,
l-a:
V, log
Ez=ı exp( - (x((1))
of > x sampled iid from Nloo
Ito I)
end for return XD
(a)
Diffusion sampling process
XT
Xt
Xt-1
Xo
Xar
Xan
Xay
Xon
Xou
the 1-th step
Vx (xx(4)()))))
Vog f(x(i)
X1
X
X1-1
LGG
Xe
X
Xi-1
lagttom
Vx logf(X(4)
FreeDoM
X1
X
X
logtoe
logfors
iterate
Vxx (20
MPGD
X1
X--
UGD
X7
Xi-1
Xi-1
+ Vxn
+Xx
Via (xxg(
Xan
Xay
Xan
Xa
X111
iterate
Figure 8: (a) The reversed diffusion process. (b) Illustration of different training-free free guidance algorithms the t-th reversed diffusion step.
21

---

C
Proofs
We prove Theorem 3.2 and Lemma 3.3 below.
C.1 Proof of Theorem 3.2
Proof. For each algorithm, we prove the equivalence of design space separately below. Notice that when 7 = degrades back f.
MPGD (Algorithm 3). Below we demonstrate that any hyper-parameter fc in Algorithm 3 is equivalent the TFG with ff (Zoje) exp( (z y)} and saper-parameter
Nrecur 1, .iter 1, 0, p 0, (cI.... .C)
To show this, notice that since both p and ς are zero, Line 4 and Line 7 take no effect. When using the identical sampling function (Line 9), TFG generates -;-1 using
-t-1 = Sample( ( Zojz, t) + c a log (2o)))
04-1 of 2t v Vat + +ott Ct C1 Vmage log (foj))
130
V1 - āt
=Vo--1(a00 ca V mojs Maroje v))
+ v ū:-1 ofc0 O1tss
which is exactly the formula used in MPGD.
DPS (Algorithm 2) and FreeDoM (Algorithm 4). We prove both algorithms together DPS is a special case of FreeDoM (without recurrence). Specifically, any hyper-parameter (p: } time travel step r and distance function D(c,,) in Algorithm 4 is equivalent to TFG with f((oo)) exp(-D(c, 2 and hyper-parameter
Nrecur Naer 0, 0,p (Vaipı, VaTTT) 1 0.
To show this, notice that both algorithms have the identical resampling step from 2;-1 to Dt, so it suffices to prove that the formula to generate It-1 in each recurrent step is the same. For FreeDoM, we have
Sample(zz, Zojt t) p V., D(c, Øoje) Øt-
Sample(z,, Tujt, t) (Vorst)) V æ, log f( yau.
and the last line equals the combination of Line 7 and Line 7 in TFG.
LGD (Algorithm 6). Any hyper-parameter {(:)T=,, n in LGD is equivalent to TFG with f(x) = exp(-ly(at ),)), sample size in Line 4, and hyper-parameter
Nrecur = 1, Niter 0, 1, 0. = (S....,, ,
Notice that ar in TFG equals 1 1/(1 + af) in the DPS algorithm. With this, the equivalence is clear from the pseudo-code of both algorithms.
UGD (Algorithm 5). exp[-l(c,f(an) f(x))) and
Any hyper-parameter k, m, s(t) in UGD equivalent to TFG with f(x)
Nrecur = k. Nuter m. =0,
p= (- -Vars((()) H -= (-
a1 51
8r)
a1
āT
where ó; is the coefficient e (x,, t) in sampler S in the UGD algorithm (which is negative). To show this, notice that A, = Pt V.., log f( -P: V.n. E(c, f(x :)) = (a, V {(c, f(zo)))) By replacing this into Line 9, the equivalence of guidance Variance Guidance can be easily observed. A similar deduction can be made for the mean guidance as well.
22

---

C.2 Proof of Lemma 3.3
Proof. Recall that Z0j:
According a simple chain rule, we have
Vās
A = Pt Vn, log ((zo
(9)
I vI ai V ca t) V mojs log ((
Pt
(10)
Vān
The perfect optimization assumption implies the relationship between co and the score of pe(a:)) which we leverage and obtain
I ai
Co
co
A = Pt
V Voii
log f f(z0)))
(11)
Vāx
I + (1 at)
a V ai logp: (3:)
Pt
Vān
V 2011
log fx
(12)
Thus, it remains to draw a connection between the conditional covariance between En and 2 log (at.). Omit the subscript æ; in Vay, we have
(ate (og))( (Vlog)((s))
2 (ox)
(13)
(x(e)))
(o( (20 (oggm(())[Vl)ogpmma))
(14)
(15)
Notice that
v ((pt((a;(00)) = Prio log + Pajol ( x0))
(2)) (zo))
Ppo(az I (1p(( 2t v /a.00 )( It v /0,00
1 Or.
1 dr
Replacing the LHS in the above equation in Equation (14), and noticing that [[ m (1+ (11-),)))) V log pog.) we have Vau
v Vara00
Bt. v (0,000 dæz
2 logg:((:))
I at
Poje (ojo [x)
āe
V log Pa (xx) )(
Pojr xx) F1 Vāre0
F. V yT dxx
I+
āi
- αe.
1 - ā
[r1] - 3t VO
[c:]]
20
1-
āe
1
I Cov Vare0 Dt x
āi
- ā.
Ōt Eojr-
I + (1 02)
āi
The proof is finished by substituting 2 log (x(;) in Equation (12) by the above result.
23

---

D
Task details
D.1 Gaussian Deblur
In computer vision, the Gaussian deblur task addresses the challenge of restoring clarity to images that have been blurred by Gaussian process. Gaussian blur, common image degradation, simulates effects such as out-of-focus photography or atmospheric disturbances. It is characterized by the convolution of an image with Gaussian kernel, process that spreads the pixel values outwards, leading to a smooth, uniform blur [33]. The deblurring task seeks to reverse this effect, aiming to retrieve the original sharp image.
Guidance target. Specifically, we apply 61 61 sized Gaussian blur with kernal intensity 3.0 to original 256 256 images. A random noise with variance of x² 0.05² added to the noisy images. If we denote the above process as a blurring operator Ablur - y. where R 2000x26x33 y E R256x 250x3 are original images and noisy images, then the target of Gaussian Deblur is to generate clean image such that:
max (@2)) = max exp(-IlAAuurraa yll2),
which means if project the generated image into the noisy space, the noisy samples (atar should be similar to the ground truth noisy images y.
Evaluation metrics. In our experiments, evaluate each guidance method set of 256 samples generated by Cat-DDPM. We apply the FID [26] to assess the fidelity, Learned Perceptual Image Patch Similarity (LPIPS) [80] evaluate the guidance validity.
24

---

Noisy images
DPS
LGD
MPGD
FrecDoM
UGD
TFG
Figure 9: Quantitative comparison of different training-free free guidance methods on Gaussian deblur task. Our TFG method can produce clean images without background noise (unlike FreeDoM and UGD), faithful image features (unlike DPS) and vivid image details (compared to LGD). Ne is set to for all methods.
25

---

D.2 Super Resolution
Super-resolution in computer vision refers to the process of enhancing the resolution of an imaging system, aimed at reconstructing high-resolution image from one or more www-reooutiion observa- tions. This technique fundamental overeoming the inherent limitations of imaging sensors and improving the detail and quality of digital images. Super-resolution has broad applications, ranging from satellite imaging and surveillance to medical imaging and consumer photography [46].
Guidance target. Specifically, we simply down-sample to original 256 x 256 images to the resolution of 64 64. random noise with a variance of o² 0.05² is also added to the noisy images, If we denote the above process R256x256x3 y E 2562x are original images and down-sampled images, then the target of super- is to generate a high resolution image Ze such that:
degradation operator Adown
y, where
max p(x0) = max exp(-=lAAdown(a yl12),
which means if we project the generated image into the downsampled image space, the downsampled samples Adown (x) should be similar to the ground truth downsampled images y.
Evaluation metries. Similar to Gaussian Deblur, we evaluate each guidance method set of 256 samples generated by Cat-DDPM. We apply FID [26] to assess the fidelity, Learned Perceptual Image Patch Similarity (LPIPS) [80] to evaluate the guidance validity.
26

---

Noisy images
DPS
LGD
MPGD
FrecDoM
UGD
TFG
Quantitative comparison of different training-free ffee guidance methods on super-resolution task. Our TFG method can produce clean images, faithful image features (unlike DPS, MPGD) and vivid image details (compared LGD, UGD). Nreur set to I for all methods.
Figure 10:
27

---

D.3
Label Guidance
Label guidance is standard task for conditional generation studied in previous literature [7, 23]. The target is to generate images conditioned on certain label. We found this standard task is rarely studied training-free guidance work and there exist an evident performance gap between training-based guidance and existing training-free guidance as shown in Section 3.
Label sets. In experiments, studied labels from CIFARIO [30] and ImageNet [55]. We average the results on labels from CIFAR10 if there is no extra explanation. For ImageNet, which resource-hugge do comprehensive inference, we randomly select labels (111, 222, 333, 444) to evaluate the methods. The corresponding label-ID and their names are as follows:
Table 6: CIFAR-10 Dataset Labels
Label-ID
Label Name
0
Airplane
Automobile
2
Bird
Cat
4
Deer
Dog
6
Frog
Horse
8
Ship
Truck
Table 7: Selected ImageNet- IK Dataset Labels
Label-ID
Label Name
111
nematode, nematode worm, roundworm
222
Kuvasz
333
Hamster
444
bicycle-buil-for-two, tandem bicycle, tandem
Guidance target. For each dataset, we use the output probability of a pre-trained classifier h(-) as the target probability. Our target is to maximize the certain classification probability of given label, i.e.,
max p(x.) max softsm(((((xx))) 200
where i is the label-ID of the target label, and h(-) is the logits of wo produced by the pre-trained classifier. For CIFARIO and ImageNet, we use pre-trained classifier based on ResNet [17] and VIT [9] that are provided from [75] and [9] respectively. The image resolution for CIFAR10 and ImageNet are 32 x 32 and 256 x 256 respectively.
Evaluation metrics. We follow the image generation literature to use Fréchet inception distance (FID) [20] assess the fidelity of generated images. The reference images are filtered from the entire dataset of CIFARIO or ImageNet with the target label, and we set sample sizes as 2560 and 256 for CIFAR and ImageNet respectively. For validity, we use another pre-trained classifier to compute accuracy other than the one used in providing guidance to avoid over-confidence:
accuracy #classified as target label
#generated samples
For CIFAR1O, pre-trained ConVNeXT [36] downloaded from HuggingFace Hub?, And for ImageNet, we use the pre-trained DeiT [69] downloaded from HuggingFace Hub".
"nctpe:/hhugiingaan co co/ahsanjavae wetpenjjaiicccoooeecece convnext tiny- certrttttin---i-iieeeeeee finetuned cifar10
fattpe://hugginngaace http://aw....eocccccoooo////e/ smal--malaallll- wawalllpptchh6-220 224
28

---

DPS
LGD
MPGD
FreeDoM-4
UGD-4
TFG-4
Figure 11: Quantitative comparison of different training-free free guidance methods on ImageNet label guidance (with target = 222, Kuvasz). The suffix of FreeDoM, UGD, TFG represents the number of recurrence Nrcur- Notice that all the samples are generated based on the same seed and we do not conduct cherry-picking. It is apprent that TFG generates the most valid samples among all the compared methods.
29

---

D.4
Combined Guidance
An interesting scenario for conditional generation is assign multiple targets for a single sam- ple. Conditional generation with multiple conditions is crucial in machine learning for enhancing the relevance and applicability of Al across complex, real-world scenarios. enables models to produce more contextually appropriate and personalized outputs, crucial for fields requiring high customization.
Motivations. Combined guidance is to use multiple target functions to guide the same sample towards multiple targets for the same sample. It is more efficient for training-free guidance to do combined guidance as the space of combinatorial targets is potentially huge, which makes it unrealistic to train all target combinations for training-based guidance. We also find it related to the topic of spurious correlations [60], where certain combinations of attributes may dominant the other combinations. For example, hair color may have a strong correlation with gender in CelebA dataset [37]. It is beneficial to explore training-free guidance reducing the bias of generation models trained on these biased data and address the concerns related to fairness and equality.
Guidance target. We study combined guidance CelebA-DDPM, which is trained CelebA- HQ [26] dataset. The image resolution is 256 256. We choose two settings of combined guidance with two attributes: (gender, hair color) and (gender, age). Each attribute has two labels, where gendere (male, female), agee {young, old]. and hair colore (black, blonde}. We have a binary classifier for each attribute that is downloaded from HuggingFace Hub The target is to sample images that maximize the marginal probability:
max Peombined (30) = max Puagett (harenn))))) 15g
where (aag)((o)) is computed using label guidance as shown in Appendix D.3.
Evaluation metries. As it is hard to filter many reference images for combined targets in CclebA- HQ dataset, we adopt Kernel Inception distance (KID) [5] to assess fidelity of generated samples using 1,000 random sampled images of CelebA-HQ as reference images. We generate 256 samples for each evaluated method. We follow MPGD [18] to use the logarithm of KID, i.e. KID (log). For validity, we use another three attribute classifiers compute the accuracy considering the conjunction of attributes:
# Atarget label (classified as target label) #generated samples
accuracy
The evaluation classifiers also downloaded from HuggingFace Hub 121314
/ http:///www.....aaeeaa/aaaaaaaaa
'Age:
'Genndr: http://wwwgggttt....o.//raaaaaaa
Haiir color: http::///wgggggg.........eeettts
12 Age (Evaluation): bbamooaatt/sssii-aeeeaaaaaaiiirr
Genner
'gg (Evaluation): https: http:////////ggggoooooooo//////a
30

---

DPS
LGD
MPGD
FreeDoM
UGD
TFG
Figure 12: Quantitative comparison of different training- free guidance methods on combined guidance task ymale+young). Our TFG method can produce images with high fidelity and validity compared to all the baselines. Notice that we use the fixed seed for all the methods in this figure and do not conduct cherry picking. Neeur is set to for all methods.
D.5 Fine-grained Guidance
Fine-grained classification a specialized task computer vision that focuses on distinguishing between highly similar subcategories within larger, general category. This task is particularly challenging due to the subtle differences among the objects or entities being classified. For example, in the context of animal species recognition, fine-grained classification would involve not just distinguishing a bird from a cat, but identifying the specific species birds, such as differentiating between crow and raven [79].
Studying fine-ge -raiier generation in the context of generative models like Stable Diffusion or DALL-E presents unique challenges due to the inherent complexity of generating highly detailed and specific images. Fine-grained generation involves creating images that not only belong to broad category but also capture the subtle nuances and specific characteristies of narrowly defined subcategory. For example, generating images specific dog breeds in distinct poses or environments requires the model understand and replicate minute details that distinguish one breed from another.
31

---

Motivations, To develop personalized AI, it is important to explore if the foundational generative models can synthesize iinegrained, accurate target samples according to user-defined target. However, this usually requires high-quality and detailed training data, and the model should be highly sensitive small variations in input to accurately produce the desired output, which can he difficult for strong text2image generation models DALL-E or Imagen" 16, We first study this problem in a training- free guidance scenario.
Guidance target. We study the which learns the generation of some species of birds but not comprehensively. We use an EfficientNet trained to classify 525 fine-graines bird species downloaded from HuggingFace Hub. The classifier trained on Bird Species dataset on Kaggle 18, We follow the same way in label-guidance to maximize softmax probability for target fine- iineggriandd label. We randomly sample 4 labels (111, 222, 333, 444) Bird Species dataset, which are:
fine-grained label guidance on ImageNet-DDPM,
Table 8: Selected Bird Species Dataset Labels
Label-ID
Label Name
111
Brown headed cowbird
222
Fairy tern
333
Lucifer hummingbird
444
Scarlet macaw
Evaluation metrics. Similar to label guidance, we use FID to evaluate generation fidelity by filtering data of the target label reference images. We also compute the FID with 256 sampled images. For accuracy, we adopt another downloaded pre-trained classifier trained on Bird Species dataset from HuggingFace Hub. 19.
32

---

Figure 13: The sampled 256 images for fine fininne guidance with target label 222 (fairy tern) by ImageNet-DDPM with TFG. A key feature of fairy tern is its black-colored head. We observe that TFG generally samples images with black, round shapes and successfully generates birds with target feature (red circled).
D.6
Style Transfer
Style transfer is significant task in computer vision (CV) that focuses applying the stylistic elements of one image onto another while preserving the content of the target image. This task is pivotal because it bridges the gap between artistic expression and technological innovation, allowing for the creation of novel and aesthetically pleasing visual content. Applications of style transfer vast, ranging from enhancing user engagement in digital media and advertising to aiding in architectural design by visualizing changes in real-time.
Guidance target. The target in our experiments guide text-to-image latent diffusion model Stable-Difuusoonv-115 [53]20 to generate images that fit both the text input prompts and the style of the reference images. The guidance objective involves calculating the Gram matrices [25] of the intermediate layers of the CLIP image encoder for both the generated images and the reference style images. The Frobenius norm these matrices serves as the metric for the objective function. Specifically, for reference style image He and decoded image D( generated from the
20 http://www..e.eee////////tteeeee diffullleeeffffffoooooo v1-5
33

---

estimated clean latent variable Zojt, compute the Gram matrices G( and ((D(z))))) These matrices are derived from the features of the 3rd layer of the image encoder, accordance with the methodologies described in MPGD [18] and FreeDoM [78]. The target function is computed as follows:
max Panyle (xo) max exp exxaxe G(D(Z))))) 85g
abo
where || 1 11 denotes the Frobenius norm of matrix.
Evaluation metries. We use Style score and CLIP score assess the guidance validity and fidelity, respectively. For reference style images and text prompts, we select 4 images from WikiArt [56] that are also used by MPGD [18], and 64 text prompts from Partiprompts [77]. For each style, generate 64 images based on the 64 different text prompts. To avoid over-confidence of CLIP score, we use two different CLIP models downloaded from HuggingFace Hub compute guidance and evaluation metrics, respectively." 2122 The style images and examplar prompts are shown as follows.
Figure 14: Four style images used in style transfer task.
Table 9: Examples for used prompts for style transfer tasks. Content Description
Content Description
book with the words 'Don't Panic!' written on it
Ground view of the Great Pyramids and
Sphinx on the moon's surface, Earth in the sky
canal Venice
white towel
Portrait of tiger wearing train conductor's 's hat and holding a skateboard Background pattern with alternating roses and skulls
Downtown Shanghai sunrise, detailed ink wash
smiling sloth holding a quarterstaff and book, VW van parked on grass
pickup truck
Concept of energy
h A shoe with sock draped over it
kitchen without refrigerator
Times Square during the day
A squirrel
turkey walking in the kitchen
bowl
The Statue of Liberty in Minecraft
A man with wild hair looking into crystal ball
Concentric circles
fire hydrant with graffiti on it
21 Guuidancee: http::/hhwggnitaaeecooo//ppaapps
Evallaatio::
34

---

Target style
hoyo
DPS
LGD
MPGD
FrecDoM
UGD
TFG
Figure 15: Quantitative comparison of different training-free free guidance methods on style transfer task with the target image as The Starry Night by Von Gogh. Our TFG generates the images with the most faithful style, while DPS, LGD, FreeDoM, and UGD fail to capture the target style. The images of MPGD also of good quality, but the style score is also inferior than TFG by large margin. We Ne = 1 for all methods.
35

---

D.7 Molecule Property Guidance
Setup. Our benchmark setup generally follows [24, 3] but with certain specifications guarantee the overall framework abides by the training-free regime. For dataset, we employ QM9 [50] and adopt the split in [24] with 100,000 training samples. Following [24] and [3]. the training set further split into two halves that guarantees there is no data leakage. The first half is leveraged to train property prediction network with EGNN [57] as the backbone, which serves as the ground truth oracle to provide the label used for MAE computation. We reuse the checkpoints released by [3] for the labeling network for all 6 properties. The second half is used train the diffusion model as well as the guidance network. We adopt the unconditional generation version of EDM [24] as the diffusion model. The guidance network general takes the same architecture as defined by [3] that, again, features EGNN as the backbone but outputs a single scalar the predicted quantum mechanics property. The only difference lies at training time we mask the diffusion time step by zeros and always the original clean molecule structure input, ensuring training- -freengggeeeeeeee objective. All the pretrained models are trained separately for different properties. At sampling time, we employ DDIM [62] sampler with 100 sampling steps, as opposed to [24, 3] that take 1000 sampling steps.
Guidance target. We study training-fres guided generation of molecules on 6 quantum mechanics properties, including polarizability (a:), dipole moment (μ),, heat capacity (C,), highest orbital energy (еhOмо), lowest orbital energy (eLUMO) and their gap (Ac). Denote the oracle property prediction network as E. Our guidance target in this case is given by
f(z, c) = exp( (esp(-IEE(e) (x() cll2).
(16)
where æ is the input molecule and c the target property value.
Evaluation metrics. We use Mean Absolute Error (MAE) and validity our evaluation metrics. In particular, MAE is computed between the target value and the predicted value gathered from the labeling network. Validity is computed by RDKit which measures whether the molecule is chemically feasible. We generate 4096 molecules for each property for evaluation.
36

---

Polarizability (a)
40
60
80
100
1
TFG (Ours)
DPS
35
LGD
C3
FreeDoM
MPGD
UGD
Figure 16: Quanlitative comparison of different maining-free guidance methods molecule genera- tion task with the target property a (polarizability). Our TFG generates valid molecules with better design target, while baselines often fail produce valid molecules offer poor guidance towards the design target. The molecules generated by our approach are increasingly polarizable as a goes up.
37

---

D.8
Audio Declipping
Audio declipping task digital audio restoration where distorted audio signals are repaired. Clipping occurs when the amplitude or frequency of an audio signal exceeds the maximum limit that the recording system can handle, leading harsh, distorted sound with portions of the waveform "cut off." Declipping aims to reconstruct the missing parts of these clipped waveforms, restoring the audio's 's original dynamies and reducing distortion. This process improves the quality and clarity of the audio, making it more pleasant to listen to while preserving the original sound's integrity.
Guidance target. Specifically, we apply distortion operation zero out the high frequency and low frequency (for the highest and lowest 40 dimensions) signal in the space of mel spectrograms, we denote the above process as a blurring operator Abtur a -> y. where a E 2256x256 y 2556x25 are mel spectrograms and noisy mel spectrograms for 5s audios, then the target of Audio Declipping to generate a clean audio Ty such that:
max p( max exp(-I]Aotur(za) -yl12 ab
ata
which means if we project the generate mel spectrogram into the noisy space, the noisy samples (z) should be similar to the ground truth noisy mel spectrogram
Evaluation metries. In our experiments, we evaluate each guidance method on a set of 256 samples generated by Audio-diffusion. We apply the Dynamic time warping (DTW) [44] to assess the guidance validity, and Fréchet Audio Distance (FAD) [28] to assess the generation fidelity.
38

---

D.9 Audio Inpainting
Audio inpainting is digital audio restoration task that involves filling in missing or corrupted segments of an audio signal. Similar to image inpainting in the visual domain, this technique reconstructs the missing portions of sound by analyzing the surrounding context and seamlessly restoring the lost information. Applications of audio inpainting range from repairing damaged recordings to reconstructing gaps audio streams due to data loss. The goal is to produce natural-sounding result that preserves the continuity and overall quality of the original audio.
Guidance target. Specifically, we apply a distortion operation to zero the middle 80 dimensions in the space of mel spectrograms. If we denote the above process blurring operator Abtur y. where æ E 255xx256 y 22266256 are mel spectrograms and noisy mel spectrograms for audios, then the target of Audio Declipping is to generate a clean audio such that:
max p())) = max exp( (ls(tl()) yl12.
which means if we project the generate mel spectrogram into the noisy space, the noisy samples (b)u((a)) should be similar the ground truth noisy mel spectrogram y.
Evaluation metries. In our experiments, we evaluate each guidance method on set of 256 samples generated by Audio-diftusion. We apply the Dynamic time warping (DTW) [44] to assess the guidance validity, and Fréchet Audio Distance (FAD) [28] to assess the generation fidelity.
39

---

E Experimental Details
E.1 Details of Table 1
In Table I, we study the effect of Monte-Carlo sample sizes in estimating the expectation of Line 4 in TFG algorithm. As the noise is added both Mean Guidance (An) and Variance Guidance (A,), decouple the effect into adding noise solely on Ao (Mean only) or A, (Variance only). In the setting of Variance only, we set 44 = 0, Nuer 0, Nevur 4, spt() "increase", and pick the best and via hyper-parameter search. In the setting of Mean only, we set p = 0, Niter = 4, Necuu = 1, 8 (t) = "increase", and pick the best μi and 5 via hyper-parameter search. We found that the sample size used in Monte-Carlo method play a neglect-able role on the performance if we set the optimal hyper-parameter. It is also noteworthy that the Monte-Carlo sampling does affect the performance of generated quality. For example, we can find that different targets shown in Appendix E.3 have different searched 5. This indicates that the best y for many targets are apparently not zero.
E.2 Comparison with grid search
We compare the performance of our beam search parameters with the full grid search ones on CIFAR- 10 label guidance task (Table 10), Overall, the performance of both search methods is identical, while grid search is much slower than our search strategy, indicating that our beam search strategy is effective and efficient.
Table 10: The searched (p, jr. 7) of exhaustive grid search and our beam search strategy on the CIFAR-10 10 label guidance task. We show the validity metric of the corresponding results and the gap ||laaliity
validity una |l. Overall, the performance of both methods is identical.
Turget
Ang.
(1.200001)
(0.25.200001)
(2.0.25.1)
(4.55.11))
(1050001))
(2.025,0000)
(0.25....1)
(1.85000)
(10:250.00))
(0.5.2.0000)
salidit
40.4
35.36%
28.25%
56.32%
20.57%
41.70%
52.56%
42.14%
83.35%
73.22%
52,50%
(8 D. 30ea
(12200001)
(0200220000)
(2.0.25.1
(402250011)
(1050001))
(200250000))
(0.25.5.11)
(1885000))
(10225000))
(05.2.0000)
80.449
35.0855
28/2516
52.81%
29,578
41.70%
51.66
42.14%
K3.8%
73.22%
51,9516
0U005
0.006
3.00%
0.00R
GL006
0,00%
0.00%
0.00%
E.3 Detailed results of each target and hyper-parameters
In this section, we present the hyper-parameee searched via the strategy introduced in Section 4 and the corresponding experimental results for TFG as shown in Table 1 I. We list several observations below.
Overall, optimal parameters vary widely between problems and datasets. For example, even with the same model and objective (e.g., label classifier ImageNet CIFAR10), the best hyperparameters vary widely from target to target. This highlights the importance of hyperparameter search.
The improvement of TFG over existing methods depends heavily on the difference between the optimal parameters and the subspaces of existing methods. For example, the for UGD is the same as TFG for gender-age guidance task, where TFG only has 0.133% validity improvement over UGD. On the contrary, their values differ on the fine-gree fine-grained classification task, and TFG has an 18.7% validity improvement over UGD. Overall, we suppose this depends on whether the optimal parameter lies in the subspace that existing methods can find.
Though the basclines mentioned in our paper should be a special case of TFG, the results for the highest MO energy guidance in Table 3 show that MPGD outperforms TFG slightly. We want to point out that the reason TFG could occasionally have slightly worse performance in practice is due to the beam search computation limit we currently pose. More specifically, we allow TFG to search at most six steps (for all byper-parameters) and all other methods for seven steps (in their subspaces). For the MO energy task, the searched parameter for MPGD is that f = 0.016 (this is the only parameter that we need to search for MPGD), where the best (and last step) of TFG is that ğ 0.004 (because it uses one step double another parameter). If we allocate more computational budget for the beam search steps, TEG will outperform MPGD this target (in fact, eight steps suffice).
40

---

Table 11: The parameter (p. f. ?) selected by beam search strategy for all methods, tasks, and targets. The search space of each method can be found in Section 3.1. For the detailed semantics of each task, please refer Appendix
DPS
LGD
MPGD
FreeDoM
UGD
TFG
Target
P
P
P
7
P
7
CIFAR-10 label guidance
0
1
0
2
0
0
2
2
0
0.001
8
0
16
0
1
0
4
0.5
0
0
4
4
0
0.25
0.001
0
1
0
0.25
0
0.25
0.25
0.25
1
1
0.5
0
0.25
0.001
4
0.25
0
0.25
0
4
4
0.25
0.001
1
0
0
0.5
16
0
4
4
O
0.25
0.5
2
0
0.5
0
0.5
0.5
0
4
0
0.5
0.001
2
0
0
0.25
0.001
ImageNer label guidance
2
0
8
0.1
222
2
0
0.25
0.5
0
88
2
2
0.5
0.1
333
2
0
0
0.25
0.25
0
8
0
1
0
0.1
Fine-grained guidance
0.25
0.25
0
0.25
0.25
0.25
0.25
0.01
222
0.25
0
0.25
0.5
0
4
4
0
0.5
0.5
0.01
333
0.25
0.25
0
0.5
0.25
0
4
4
0
0.5
0.5
0.01
444
0.25
0,25
0
0.25
0
1
0
0.5
0.5
0.01
Combined Guidance (gender & hair)
(0,0)
0,25
0
16
16
0.01
0.01
(1.0)
4
0
16
0.25
0
4
4
0
0.01
(1.1)
2
0.25
0
8
O
2
2
D
0.5
0.1
Combined Guidance (gender & age)
(0.0)
8
0.25
0
0.25
0
1
1
0.01
(0.1)
0
0.5
0.5
0.5
0.5
0.01
(1.1)
2
0.25
0
16
0.25
0.5
0.1
Super-resolution
16
D
16
0
1
0
16
16
0
8
8
0
0.01
Gaussian Deblur
16
0
16
16
0.01
Style Transfer
0
0.25
0.25
0.01
4
0.25
0
4
0.5
0
1
0
0.25
0.1
2
0
0.25
0
8
0.25
0
1
1
0
0.25
0.1
2
0
0
2
0.25
0
0.25
0.25
0
0.25
0.01
Molecule Property
0.02
0.005
0.005
0.005
0.001
0.002
0.1
0.005
0
0.005
0
0.005
0.005
0.005
0.005
0
0.004
0.001
0.001
0.005
0
0.005
0
0
0.01
0.005
0
0
0.005
0.005
0
0.002
0.004
0.001
0.005
0
0.01
0
0.01
0.005
0
0.005
0.005
0
0.016
0.002
0.0001
0.005
0.01
0
0.01
0.01
0
0.005
0.005
0.032
0.001
0.001
Audio Declipping
0
16
0.1
Audio Inpainting 16
16
0
16
0
16
16
0
0.25
0.1
E.4 Tricks implemented in FreeDoM codebase
In the codebase of 2 the schedule of guidance strength for different applications different. For example, the guidance strength has a schedule coefficient Vāz for face generation, and the schedule for style transfer is complex and involves correction term, the mean of gradients' norm, and specific constant coefficient 0.2. The paper does not mention this particular schedule, leaving
23
41

---

the rationale for choosing these schedules unclear. We choose not to include the tricks and find that with our unified hyper-parameter searching strategy. the performance of FreeDoM is similar.
E.5 Hardware and Software
We run most of the experiments on clusters using NVIDIA A100s. We implemented our experiments using PyTorch [49] and the HuggingFace library. 24 Overall, estimated that total of 2,000 GPU hours were consumed.
hhttpe://uugiingaacc.co
42

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have both theoretically and empirically justified our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
43

---

Justification: See Sec. C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See Sec. D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
44

---

Answer: [Yes]
Justification:
See supplementary file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wttpccccpppcccaeecccccccccccctlt / sssssssoiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: See Sec. D.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiment is conducted extensively large scale, and existing works the same line were not reported since the numbers are stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
45

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttprr//////e../eeeeeeeeee cc/ ppublic/Ethi cttpppiieeciccccuueeeeeeeeeeeeee csQuidel ines?
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Sec. 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
46

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
It is not applicable to the concern of this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the related papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
47

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not include new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do involve human subjects and studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
48


Reviews:
[{"review_id": "swiPXPP7r6", "reviewer": "Reviewer_DCpp", "summary": "This paper focuses on the unification of training-free guidance methods for diffusion models. It defines each method within a unified framework and finds that restricting the hyperparameter space is consistent with existing methods. This framework can be broadly categorized into mean guidance, variance guidance and implicit dynamics. The paper demonstrates performance improvements in several experimental scenarios.", "questions": "Please see the Weaknesses part.", "limitations": "They provided in the last section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Experiments were conducted on different data sets and scenarios.\n* Training-free guidance, which had been developed in different ways, was unified into a single framework.\n* If the code is open-sourced, it will greatly benefit the diffusion community.", "weaknesses": "* Algorithm 1 seems to be one of the most important parts of this paper, but it lacks a detailed explanation. An explanation of Algorithm 1 along with the corresponding hyperparameters from Definition 3.1 in Section 3.1 would greatly aid understanding.\n\n* In Algorithm 1, the iteration part in line 8 could be explicitly described using a for loop in the algorithm, rather than as a comment.\n\n* I believe that Theorem 3.2 and its proof are not well formulated mathematically. If the authors want to formalize it as a theorem, they need to show mathematically that the same generated distribution can be produced. In my opinion, an explanation at an analytical level would be acceptable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful and constructive review, and we are delighted that you think our work will greatly benefit the diffusion community. Regarding to the concerning questions:\n\n> Algorithm 1 seems to be one of the most important parts of this paper, but it lacks a detailed explanation. An explanation of Algorithm 1 along with the corresponding hyperparameters from Definition 3.1 in Section 3.1 would greatly aid understanding.\n> \n\nWe agree that more explanation w.r.t. Alg 1 and the hyperparameter space will help with the understanding. We will incorporate the explanation of the relation between Alg 1 and Def 3.1: generally speaking, Alg 1 contains three operations (Mean Guidance, Variance Guidance, and Implicit Dynamics), and each operation in the algorithm separately control part of the hyper-parameter in Def 3.1. We will explicitly point how this relations, and how each operation in the algorithm affects the parameter space, i.e., what part of the space is controlled. \n\n> In Algorithm 1, the iteration part in line 8 could be explicitly described using a for loop in the algorithm, rather than as a comment.\n> \n\nWe agree that explicitly writing down the for loop will make it more clear. Thanks for the suggestion.\n\n> I believe that Theorem 3.2 and its proof are not well formulated mathematically. If the authors want to formalize it as a theorem, they need to show mathematically that the same generated distribution can be produced. In my opinion, an explanation at an analytical level would be acceptable.\n> \n\nWe really appreciate the suggestion, and we believe that keeping it as a theorem would still be helpful. That said, we will make the definition and proof more mathematically formal. Specifically, we will explicitly define the parameter space of each of the existing algorithms, and pointing out that each instantiation in their distribution corresponds to which parameter value in our parameter space.\n\nAgain, we sincerely thanks for the helpful review you provide. Please let us know whether there is any additional concern that we could address to help improve your evaluation to our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OqNHnh1EsK", "reviewer": "Reviewer_J9FB", "summary": "This paper scrutinizes existing works on training-free guidance in diffusion models and proposes a unified framework that includes all existing methods as special cases. With this unified framework, this work presents a detailed and informed investigation of the design choices and hyperparameters within this framework. Additionally, it proposes a comprehensive benchmark involving 14 task types and 38 targets to evaluate the performance of this unified framework by optimizing the hyperparameters in the design choices.", "questions": "1. Why does the highest MO energy in Table 3 show a negative improvement? MPGD should be a special case of TFG. Is this because the grid-searched hyperparameters for TFG are not optimal?\n\n2. Can the author comment on other lines of work [1,2,3,4], specifically direct optimization approaches, on training-free optimization of diffusion models with a target objective? The task setting is the same as training-free guidance. Although it is evident that the direct optimization approaches are slower than training-free guidance, it is not clear how their performance compares.\n\n[1] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7280–7290, 2023b.\n\n[2] Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. arXiv preprint arXiv:2402.14017, 2024.\n\n[3] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. arXiv preprint arXiv:2312.11994, 2023.\n\n[4] Tang, Zhiwei, et al. \"Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization.\" arXiv preprint arXiv:2405.18881 (2024).", "limitations": "See my comments on the weaknesses above. I think there are a few unclear points worth discussing.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed unified framework is a very interesting and novel summarization of the methods in existing works. Existing training-free guidance often uses different notations and different ways to formulate the problem. It is very helpful that this framework elucidates the design choices in training-free guidance.\n\n2. The proposed benchmark is very comprehensive, and I believe it will certainly be helpful for the community to continue conducting more research on this topic.", "weaknesses": "1. While I appreciate the comprehensiveness of the experiments on the proposed benchmark, I think the author fails to provide an informative analysis of the results. Currently, the results simply show that by optimizing the hyperparameters in the unified framework, we obtain better performance, which is a natural outcome since the existing methods are included in the framework. From a research perspective, I can think of several questions worth investigating, such as whether the optimal hyperparameters vary between tasks and, when using fixed models, how the optimal hyperparameters are affected by the target objective function. With this investigation, it would be best to reach some general conclusions to guide users in tuning the hyperparameters in practice, rather than just relying on grid search.\n\n2. It seems that for around half of the tasks, the improvement is only marginal, while for a few other settings, the improvement is significant. I think this phenomenon is worth further investigation to understand in what scenarios TFG will provide improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thanks for your insightful and constructive review, and we are more than honored that you think our work is interesting, novel, comprehensive, and will be helpful for the community. Regarding your concerns,\n\n> While I appreciate the comprehensiveness of the experiments on the proposed benchmark, I think the author fails to provide an informative analysis of the results. Currently, the results simply show that by optimizing the hyperparameters in the unified framework, we obtain better performance, which is a natural outcome since the existing methods are included in the framework. From a research perspective, I can think of several questions worth investigating, such as whether the optimal hyperparameters vary between tasks and, when using fixed models, how the optimal hyperparameters are affected by the target objective function. With this investigation, it would be best to reach some general conclusions to guide users in tuning the hyperparameters in practice, rather than just relying on grid search.\n> \n\nWe agree the importance of more fine-grained analysis, and we are delighted to investigate more on our experimental results as you suggested. Specifically, we first explicitly present the optimal hyper-parameters of TFG and other existing methods for all tasks in the uploaded PDF (Table 2). We list several observations below.\n\n1. Overall, optimal parameters vary widely between problems and datasets. For example, even with the same model and objective (e.g., label classifier on ImageNet or CIFAR10), the best hyperparameters vary widely from target to target. This highlights the importance of hyperparameter search.\n2. The improvement of TFG over existing methods depends heavily on the difference between the optimal parameters and the subspaces of existing methods. See the next questions for a more detailed analysis.\n3. Also note that in our paper, we didn’t do an exhaustive grid search, but instead perform an efficient beam search (line 254-264) to find the optimal hyper-parameters. All searches are run with 1/8 of the test sample size and a maximum search step of 6 (line 287). Our experimental results demonstrate that this is sufficient to find near optimal parameters (check out Table 1 in our uploaded PDF).\n\n> It seems that for around half of the tasks, the improvement is only marginal, while for a few other settings, the improvement is significant. I think this phenomenon is worth further investigation to understand in what scenarios TFG will provide improvements.\n> \n\nThanks for the advice! We conducted a detailed investigation into the phenomenon and found that the improvements are highly related to the differences in the optimal hyperparameters between TFG and existing methods. For example, the $\\bar\\rho$ for UGD is the same as TFG for gender-age guidance task, where TFG only has 0.133% validity improvement over UGD. On the contrary, their values differ on the fine-grained classification task, and TFG has an 18.7% validity improvement over UGD. Overall, this depends on whether the optimal parameter lies in the subspace that existing methods can find. We will add this analysis to our revised manuscript. \n\nIn addition, regarding to the questions,\n\n> Why does the highest MO energy in Table 3 show a negative improvement? MPGD should be a special case of TFG. Is this because the grid-searched hyperparameters for TFG are not optimal?\n> \n\nWe appreciate the reviewer for carefully examining our paper, and we want to point out that the reason TFG could occasionally have slightly worse performance in practice is due to the beam search computation limit we currently pose. More specifically, we allow TFG to search at most six steps (for all hyper-parameters) and all other methods for seven steps (in their subspaces). For the MO energy task, the searched parameter for MPGD is that $\\bar \\mu = 0.016$ (this is the only parameter that we need to search for MPGD), where the best (and last step) of TFG is that $\\bar \\mu = 0.004$ (because it uses one step to double another parameter). If we allocate more computational budget for the beam search steps, TFG will outperform MPGD on this target (in fact, eight steps suffice). \n\n> Can the author comment on other lines of work [1,2,3,4], specifically direct optimization approaches, on training-free optimization of diffusion models with a target objective? The task setting is the same as training-free guidance. Although it is evident that the direct optimization approaches are slower than training-free guidance, it is not clear how their performance compares.\n> \n\nThanks for the valuable suggestion. We review all papers to understand the direct noise optimization approach (DNO), and we believe that this DNO has a different motivation than training-free guidance. Specifically, DNO is not only slow but also GPU-memory intensive, as gradients have to be propagated through the entire ODE process for multiple times until convergence. This makes it hard to implement and study within a short amount of time. We would like to leave the comparison to future studies, and we sincerely hope you can understand the difficulty.\n\nAgain, we sincerely thanks for the helpful review you provide. Please let us know whether there is any additional concern that we could address to help improve your evaluation to our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x8WtCD2zbR", "reviewer": "Reviewer_aTDj", "summary": "The authors propose a framework (TFG) for training-free guidance of unconditional diffusion models, enabling their application to conditional generation tasks such as super-resolution, deblurring, etc. via the use of a predictor that evaluates the quality of a clean sample.  TFG, as in related past methods, aim to sample from the conditional distribution of samples defined in Equation 4 without either training a conditional diffusion model or a predictor evaluating the quality of noisy samples as in classifier guidance.  The gradient of the predictor on noisy samples combined with the unconditional score produce the correct conditional score function in Equation 5.  The challenge of training-free methods is to approximate this gradient somehow and a variety of approaches have been previously proposed.  TFG defined in Algorithm 1 is shown to include five such previous training-free methods as special cases for particular hyperparameters, demonstrating that these methods can be viewed and understood in a unified framework.  The authors then propose a procedure to optimize TFG's hyperparameters jointly.  By leveraging the larger design space with optimized hyperparameters, performance gains are observed across 14 tasks, 6 diffusion models, and 38 target predictors compared to the past methods subsumed by TFG.", "questions": "1. Why does increasing $N_{recur}$ and $N_{tier}$ eventually hurt performance?\n2. In section 4.2, does the \"structure analysis\" that \"increase\" is best hold more generally on other tasks than label guidance?   \n3. The number of Monte Carlo samples in the Implicit Dynamic is set to 1 based on Table 1, but that choice needs more justification.  Does more samples than 4 help?  Why is implicit dynamic helpful with few samples?  This seems contrary to the original motivation of estimating an average.\n4. In section 4.2, the beam search in \"Searching strategy\" could use more detail.  What sample sizes are used and how did that enable quick search? Also, what metric is used when deciding the top K candidates in this search?", "limitations": "Limitations are briefly discussed in Section 6, focusing on why training-free guidance remains relevant given language-based image generators.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The TFG framework appears to be a non-trivial, novel incorporation of recent training-free methods and helps contextualize the relationships of these methods to one another.  Training-free guidance of diffusion models is an unsolved problem with many recent papers and placing a portion of this literature into a unified framework is a valuable contribution.  \n- The framework also led to substantive performance gains on generation validity in the comprehensive benchmarks.  The benchmark evaluations included a good variety of tasks, models, and predictors, including particularly out-of-distribution fine-grained label guidance and molecule property guidance.\n- The paper is generally well-written and organized.", "weaknesses": "- Some statements and claims are overly broad.  The authors claim all existing training-free approaches fit into their framework which is unlikely.  Many training-free methods have been proposed and the authors could better place their framework in the context of recent literature.  Section 3 Figure 1's demonstration that training-based conditional generation outperforms training-free is unsurprising, given that training-free methods are approximations.\n- While hyperparameter search was helpful, the unified design space discussion in Section 4.1 led to limited theoretical insight.  The authors could consider expanding this section to improve their methodological contributions.\n- The hyperparameter setting strategy described in Section 4.2, key to getting gains versus past methods, could use more detailed explanation (see questions 2-4 below). \n- The benchmark Section 5 decides to not compare generation fidelity in favor of comparing the best algorithm in terms of generation validity.  However, throughout the rest of the paper (e.g. figure 1, 2, and 3) tradeoffs between fidelity and validity are emphasized.  Looking at Table 2, TFG is sometimes but not always the best algorithm in terms of both fidelity and validity simultaneously.  The focus on only validity seems unjustified here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful and constructive review, and we are honored that you think our work is non-trivial, novel, organized, comprehensive, substantive in experiments, and presents a valuable contribution. We are happy to address your concerns:\n\n> The authors claim all existing training-free approaches fit into their framework which is unlikely. Many training-free methods have been proposed and the authors could better place their framework in the context of recent literature. Section 3 Figure 1's demonstration that training-based conditional generation outperforms training-free is unsurprising, given that training-free methods are approximations.\n> \n\nOur algorithm is based on a comprehensive contemporary literature review of more than ten papers (see [1]-[10] below) and on a collection of all of their algorithms (that directly or indirectly use the algorithms we unify), trying to provide an elegant and unified way to study the problem. That said, we completely agree with your suggestion that we never fit all methods, and we are more than happy to explicitly clarify this and discuss our framework in the context of TFG literature. \n\nThe purpose of Figure 1 is to emphasize our observation that, unlike what previous works have claimed, training-free guidance is far from being addressed, and all existing methods could even fail on a very easy task. In other words, it helps demonstrate our motivation, rather than to help compare training-free methods with training-based methods. We will clarify this.\n\n> While hyperparameter search was helpful, the unified design space discussion in Section 4.1 led to limited theoretical insight.\n> \n\nThanks for the suggestion! Currently, we organize Section 4.1 in a way that is “less theoretical” and “more intuitive” to help readers better understand each part of the parameter space without being overwhelmed by theory concepts. That said, the study of space involves Ornstein–Uhlenbeck process, Tweedie’s formula (second order), and probability convergence. We will try to extend our theoretical insights and put more fine-grained discussions in the appendix to ensure both intuition and theory are clearly conveyed to readers.\n\n> The hyperparameter setting strategy… could use more detailed explanation (see questions 2-4 below).\n> \n\nWe answer each question separately below.\n\n> The benchmark Section 5 decides to not compare generation fidelity in favor of comparing the best algorithm in terms of generation validity. However, throughout the rest of the paper (e.g. figure 1, 2, and 3) tradeoffs between fidelity and validity are emphasized. Looking at Table 2, TFG is sometimes but not always the best algorithm in terms of both fidelity and validity simultaneously. The focus on only validity seems unjustified here.\n> \n\nWe thank you for allowing us to further explain this. We completely agree that there is a trade-off between fidelity and validity, and depending on the user’s requirements, different metrics should be emphasized. The reason why we majorly compare validity is that for each algorithm during beam search, the metric we use to select the best run is exactly the validity (in a held-out set, of course). It’s important that the selection metric and the evaluation metric are the same to avoid unfair and tricky comparisons. \n\nOn the other hand, users could also arbitrarily combine fidelity and validity into a new “metric” and conduct beam searches via this metric if they prefer. Our framework does not have any restriction on this, and, unsurprisingly, TFG will still outperform existing methods. We thank you for your question, and we would like to clarify this in the paper.\n\nAdditionally, regarding to the questions,\n\n> Why does increasing $N_{recur}$ and $N_{iter}$ eventually hurt performance?\n> \n\nIn fact, this phenomenon has been pointed out in previous works (e.g., MPGD) as well, and its underlying theory remains unclear. One observation is that if $N$ is too large, the generated images tend to be “valid” but highly unrealistic, possibly due to the amount of injected noise being too large. \n\n> In section 4.2, does the \"structure analysis\" that \"increase\" is best hold more generally on other tasks than label guidance?\n> \n\nYes. Generally speaking, we find that “increase” works best (or with a negligible gap) among all of the tasks we consider in the paper. \n\n> The number of Monte Carlo samples in the Implicit Dynamic is set to 1 based on Table 1, but that choice needs more justification. Does more samples than 4 help? Why is implicit dynamic helpful with few samples? This seems contrary to the original motivation of estimating an average.\n> \n\nWe find that more than $4$ of sample size does not help with sample quality because it reduces the stochasticity of the dynamics (think about changing the Gaussian noise in an SDE to, e.g., its expectation). We are delighted to provide more justification by adding a mathematical explanation of the role that the Gaussian noise is playing in the appendix and explaining why more samples are not beneficial. \n\n> In section 4.2, the beam search in \"Searching strategy\" could use more detail. What sample sizes are used and how did that enable quick search? Also, what metric is used when deciding the top K candidates in this search?\n> \n\nAs mentioned in line 286-287, all searches are run with 1/8 of the test sample size and a maximum search step of 6. The test sample size for each task can be found in Appendix D. To conduct exhaustive grid search, we need to run more than $125$ experiments, which are much slower than our beam search strategy. The metric used when deciding the top K candidates is validity. \n\nAgain, we sincerely appreciate your helpful review. Please let us know whether there is any additional concern that we could address to improve your evaluation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6i2pz4NB9X", "reviewer": "Reviewer_Xm6V", "summary": "The paper introduces Training-Free Guidance (TFG), a novel framework designed to enhance the generation of samples with desired properties using diffusion models, without necessitating additional model training. TFG aims to resolve the shortcomings of existing training-free methods by offering a unified algorithmic framework that simplifies the comparison and application of such methods across a wide range of tasks. By theoretically and empirically analyzing a hyper-parameter design space within this framework, the authors develop an effective strategy for hyper-parameter selection applicable to various tasks. Their comprehensive benchmarks across multiple tasks and targets demonstrate TFG's superior performance, achieving an average improvement of 7.4% over existing methods.", "questions": "1. The field of training-free guidance encompasses a wide array of studies beyond those specifically addressed in this paper (e.g. [1]-[3]). Could you elaborate on how these works relate to the scope and contributions of your paper?\n\n2. Your paper operates under the assumption that the forward model is known, leveraging training-free guidance for solving inverse problems. Given that knowing the forward model allows for the generation of a large number of samples to train a conditional diffusion model at a relatively low computational cost (e.g., 10 A100 GPU hours) and training-based approaches are much better than training-free ones, could you discuss the significant motivations of adopting a training-free approach in this context?\n\n3. The theoretical foundation of your paper seems to rest significantly on Lemma 4.1, which revisits the variance of MMSE estimator of the signal corrupted by Gaussian noise (e.g., (2.8) in [4]). This formula is widely adopted in diffusion papers (e.g., [5]). Given its established nature and previous applications, could you elaborate on how this lemma specifically contributes to the novel aspects of your framework?\n\n4. Concerning Lemma 4.1, it appears there is no assurance that the generated image accurately follows the conditional distribution, nor is there a guarantee that the loss decreases at each iteration as suggested by equation (7). Could you provide further clarification or additional theoretical support to address these concerns?\n\n5. Several techniques introduced in the paper lack direct theoretical underpinning:\n- The concept of \"time-travel\" being an Ornstein-Uhlenbeck process is intriguing but lacks a detailed derivation. Could you expand on how the theory of the Ornstein-Uhlenbeck process quantifies the benefits of time-travel in your framework?\n- The selection of hyperparameters seems not to be grounded in theory. Could you discuss the rationale behind these choices and any potential theoretical support?\n\n6. The paper posits that the study of training-free methods can be encapsulated within the hyperparameter space of your framework. However, specific hyperparameter settings critical for the performance in existing works, such as the step sizes used in Face generation (at.sqrt()) and Style transfer ((correction * correction).mean().sqrt().item() * unconditional_guidance_scale / (norm_grad * norm_grad).mean().sqrt().item() * 0.2) in FreeDoM, are not explicitly covered. Could you address the omission of these settings and their impact on the comprehensiveness of your study?\n\n7. The hyperparameter search settings for the baselines in your comparative analysis are not disclosed, raising questions about the fairness and validity of the comparisons. Could you provide more details on these settings to ensure a transparent and equitable comparison?\n\n[1] Feng, Weixi, et al. \"Training-free structured diffusion guidance for compositional text-to-image synthesis.\" ICLR 2023.\n\n[2] Chen, Minghao, et al. \"Training-free layout control with cross-attention guidance.\" WACV 2024.\n\n[3] Mo, Sicheng, et al. \"Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition.\" CVPR 2024.\n\n[4] Efron, Bradley. \"Tweedie’s formula and selection bias.\" JASA 2011.\n\n[5] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representation.\" ICLR 2024.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Given the burgeoning interest in the area of training-free guidance, the establishment of a unified benchmark as presented in this paper is a commendable contribution that holds the potential to significantly advance research in this field. The authors' efforts in this direction are highly appreciated and underscore the importance of standardized benchmarks for facilitating future developments.\n\n2. The experiments conducted in this study are extensive, reflecting a high degree of diligence and thoroughness. Such a comprehensive experimental approach is commendable and warrants recognition. Consequently, I believe this aspect of the paper merits a positive evaluation for its contribution to validating the proposed framework and its applicability across a diverse range of tasks.", "weaknesses": "There are several areas where improvements could significantly enhance its contributions. Addressing these points satisfactorily would make a strong case for elevating the paper's status to \"Accept\" or \"Strong Accept\".\n\n1. While the paper provides valuable insights into a specific line of training-free guidance, it is important to acknowledge the existence of a broader spectrum of works outside this category. This observation suggests that the title of the paper may slightly overreach, potentially implying a more comprehensive coverage than is actually presented. (Question 1)\n\n2. The paper commendably supports the motivations behind conditional diffusion in Appendix A, offering a solid foundation for its relevance. However, the rationale for focusing specifically on training-free approaches appears less extensively articulated. (Question 2)\n\n3. The authors claimed that they have theoretically grounded their unified framework. However, upon review, the theoretical underpinnings presented seem to require further elaboration to fully substantiate this claim. (Question 3, 4, 5)\n\n4. The authors claimed that \"the studies of training-free methods become the study within the hyperparameter space of our framework\". Nonetheless, it appears that the search space defined within this framework might not fully encompass the range of hyper-parameters considered in existing literature. This limitation could potentially restrict the framework's applicability or comparative analysis capabilities. (Question 6, 7)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful and constructive review, and we are so honored that you believe our work is novel, effective, comprehensive, and offers commendable contributions. We are delighted to address your concerns and questions below.  \n\n1. The existence of a broader spectrum of works and the overreach of the title.\n\n> The field of training-free guidance encompasses …a wide array of studies beyond those specifically addressed in this paper (e.g. [1]-[3]). Could you elaborate on how these works relate to the scope and contributions of your paper?\n> \n\nThanks for this valuable suggestion. The three works you mentioned are training-free methods specifically for text-to-image tasks (compositional generation [1], layout control [2], spatial control [3]). [1] and [2] exploited the cross-attention mechanism in Stable Diffusion UNet to incorporate guidance targets (noun phrases [1], layout condition [2]), while [3] leverages the diffusion features of the guidance image (with DDIM inversion) to guide the diffusion process.\n\nUnlike these works, which design task-specific guiding strategies, our paper focuses on general training-free guidance methods that can be applied to any diffusion model for any objective function. Basically, our work doesn’t focus on the design of the objective function for each task but on the universal methodology of guidance. That said, the improvement in architecture, diffusion models, and guidance targets is “orthogonal” to our work. We would like to discuss these works in our revised manuscript to clarify our scope and contributions. \n\n2. The rationale for focusing specifically on TFG.\n\n> Your paper operates under the assumption that the forward model is known, leveraging training-free guidance for solving inverse problems. Given that knowing the forward model allows for the generation of a large number of samples to train a conditional diffusion model at a relatively low computational cost (e.g., 10 A100 GPU hours) and training-based approaches are much better than training-free ones, could you discuss the significant motivations of adopting a training-free approach in this context?\n> \n\nWe sincerely thank the question that allows us to discuss the motivation of the paper further. In lots of cases when the conditions are complex, e.g. requires an image of a special dog species in certain scenarios or a molecule that has certain polarizability and synthesizability, they are extremely rare (with probability less than, e.g., $10^{-6}$) across the unconditional distribution. As such, “sampling a large number of targets for training a time-dependent classifier” itself is already impossible. However, training-free methods can effectively increase the sample rate from $10^{-6}$ to $10^{-2}$, making it possible to leverage training-based methods. In fact, we believe that one motivation of TFG is exactly to bridge towards the scenarios you describe. We will explicitly add this discussion in the paper. \n\n3. Further elaboration of the theoretical part of the work.\n\n> The theoretical foundation of your paper seems to rest significantly on Lemma 4.1, which revisits the variance of MMSE estimator of the signal corrupted by Gaussian noise (e.g., (2.8) in [4]). This formula is widely adopted in diffusion papers (e.g., [5]). Given its established nature and previous applications, could you elaborate on how this lemma specifically contributes to the novel aspects of your framework?\n> \n\nWe want to highlight that lemma 4.1 is simply used to illustrate the effect of “variance guidance” and is not used to prove our major theorem 3.2. Despite its wide use in previous works, here we use the lemma to point out that $\\Delta _t$ in Line 7 is to control the second-order information, pointing to the difference between $\\Delta_t$ and $\\Delta_0$ from a theoretical perspective.\n\n> Concerning Lemma 4.1, it appears there is no assurance that the generated image accurately follows the conditional distribution, nor is there a guarantee that the loss decreases at each iteration as suggested by equation (7). Could you provide further clarification or additional theoretical support to address these concerns?\n> \n\nWe agree that a theoretical guarantee on loss is important, but to the best of our knowledge, none of the existing training-free methods can prove that the generated sample follows the correct distribution as desired. This is not an issue of Lemma 4.1, but we believe that one crucial future direction is to provide a global-level guarantee for training-free guidance. The intrinsic difficulty here is to analyze the difference between a training-based classifier $f(x,t)$ and standard classifier $f(x)$, which are complex to be quantitatively captured. \n\n> The concept of \"time-travel\" being an Ornstein-Uhlenbeck process is intriguing but lacks a detailed derivation. Could you expand on how the theory of the Ornstein-Uhlenbeck process quantifies the benefits of time-travel in your framework?\n> \n\nWe are delighted to further explain this. For each recurrent step, we compute $x_{t-1} = u(x_t)$ where $u$ corresponds to Line 6-9 and then add a Gaussian noise back in Line 10 to obtain updated $x_t$. Together, the process becomes\n\n$$\nd x_t = \\sqrt{\\alpha_t }u(x_t) - x_t + \\sqrt{1 - \\alpha_t} \\epsilon,\n$$\n\nwhich is a typical OU process. This implies that when $N_{recur}$ goes to infinity, $x_t$ will converge to a certain distribution that is hard to compute analytically.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their insightful and constructive reviews, and we are honored that all reviewers believe that our paper is novel, beneficial, comprehensive, and well-written. Reviewer aTDj thinks the work is “a valuable contribution”, reviewer Xm6V thinks it “reflects a high degree of diligence and thoroughness”, and both reviewer DCpp and J9FB think it will “certainly be helpful for the community”.

All reviewers provide fine-grained and important feedback on polishing the paper, and we are more than willing to accept. As NeurIPS policy does not allow us to upload an improved version, we will explicitly explain how we improve the paper in each of your questions, and we promise that all suggestions will be addressed in the final paper. As all reviewers have emphasized the importance of the codebase and benchmarks, we will open-source our code, benchmarks, configurations, and existing runs, as promised in the paper.

Reviewer aTDj points out that the claim on unifying *all* works is overly broad, and indeed, reviewers J9FB and Xm6V list a few training-free guidance papers that are related but with different focus. We want to highlight that our work is based on a comprehensive contemporary literature review over more than 15 papers (see response to reviewer aTDj) and on a collection of all of their algorithms (that directly or indirectly use the algorithms we unify), trying to provide an elegant and unified way to study the problem. That said, we completely agree that we never fit all methods, and we are more than happy to explicitly clarify this and discuss our framework in the context of TFG literature. 

For the uploaded PDF, we present the actual parameter selected by our beam search strategy for each algorithm and task, and we additionally compare the performance of the beam search parameter with the full grid search one. Overall, these results demonstrate that the comparison between different methods is transparent and objective, and that our beam search strategy is effective and efficient compared with full grid search method.

Below are our detailed responses. Please do not hesitate to let us know if there is any additional concern that we can help address to help with an objective evaluation of our work.

Meta-review: This paper addresses the challenge of generating samples with desired target properties using an unconditional diffusion model and a target property predictor without requiring additional training. It critiques existing methods for lacking theoretical grounding and rigorous benchmarking, which can lead to inconsistent performance across tasks. The authors introduce a novel algorithmic framework that unifies existing methods under a broader, algorithm-agnostic design space. They also propose an efficient hyper-parameter search strategy and demonstrate its effectiveness through extensive benchmarking, showing a 7.4% average performance improvement across various models and tasks. The reviewers have been generally positive about the contribution of this work, and believe that the proposed benchmarking will be interesting to future works in the related areas.
Decision: Accept (spotlight)