Title: TFG: Unified Training-Free Guidance for Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion model

conditional generation

training-free guidance
Abstract: Given an unconditional diffusion model and a predictor for a target property of interest (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. Existing methods, though effective in various individual applications, often lack theoretical grounding and rigorous testing on extensive benchmarks. As a result, they could even fail on simple tasks, and applying them to a new problem becomes unavoidably difficult. This paper introduces a novel algorithmic framework encompassing existing methods as special cases, unifying the study of training-free guidance into the analysis of an algorithm-agnostic design space. Via theoretical and empirical investigation, we propose an efficient and effective hyper-parameter searching strategy that can be readily applied to any downstream task. We systematically benchmark across 7 diffusion models on 16 tasks with 40 targets, and improve performance by 8.5% on average. Our framework and benchmark offer a solid foundation for conditional generation in a training-free manner.

TFG: Unified Training-Free Guidance for Diffusion Models

Abstract

Given an unconditional diffusion model and a predictor for a target property of interest ( e.g. , a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. Existing methods, though effective in various individual applications, often lack theoretical grounding and rigorous testing on extensive benchmarks. As a result, they could even fail on simple tasks, and applying them to a new problem becomes unavoidably difficult. This paper introduces a novel algorithmic framework encompassing existing methods as special cases, unifying the study of training-free guidance into the analysis of an algorithm-agnostic design space. Via theoretical and empirical investigation, we propose an efficient and effective hyper-parameter searching strategy that can be readily applied to any downstream task. We systematically benchmark across 7 diffusion models on 16 tasks with 40 targets, and improve performance by 8.5% on average. Our framework and benchmark offer a solid foundation for conditional generation in a training-free manner. 1

1 Introduction

Recent advancements in generative models, particularly diffusion models [61, 21, 62, 66], have demonstrated remarkable effectiveness across vision [65, 48, 52], small molecules [74, 73, 24], proteins[1, 72], audio [35, 29], 3D objects [40, 41], and many more. Diffusion models estimate the gradient of log density (i.e., Stein score, [67]) of the data distribution [65] via denoising learning objectives, and can generate new samples via an iterative denoising process. With impressive scalability to billions of data [58], future diffusion models have the potential to serve as foundational generative models across a wide range of applications. Consequently, the problem of conditional generation based on these models, i.e. , tailoring outputs to satisfy user-defined criteria such as labels, attributes, energies, and spatial-temporal information, is becoming increasingly important [63, 2].

Conditional generation methods like classifier-based guidance [66, 7] and classifier-free guidance [23] typically require training a specialized model for each conditioning signal (e.g., a noise-conditional classifier or a text-conditional denoiser). This resource-intensive and time-consuming process greatly limits their applicability. In contrast, training-free guidance aims to generate samples that align with certain targets specified through an off-the-shelf differentiable target predictor without involving any additional training. Here, a target predictor can be any classifier, loss function, probability function, or energy function used to score the quality of the generated samples.

In classifier-based guidance [66, 7], where a noise-conditional classifier is specifically trained to predict the target property on both clean and noisy samples, incorporating guidance in the diffusion

(a)

(b)

?

TFG

UGD

08

LGD

DPS

FreeDoM-2

FreeDoM-1

MPGD

FreeDoM-4

MPGD

FreeDoM

70

+UGD-1

★

Accuracy (%)

UGD-2

training-based

UGD-4

TFG-4

LGD

★

DPS★

50

40

Search space for TFG

30

40

60

80

100

120

140

160

FID

(c)

(d)

(e)

Audio inpainting

Audio declipping

Combined guidance

Lostframes

(gender,haircolor)=(man,blond)

inpaintwithTFG

declipwithTFG

More low and highfrequency

(f)

Style transfer

g

Fine-grained generation

ApickuptruckAcanalinVeniceAwhitetowel

Fine-grained birdspieces=“fairytern"

"TheStarry Night"

(h) Quantum property guidance

Styleimage

TheStatue of Liberty

Concentric

AnElephant

Lowpolarizability

High polarizability

inMinecraft circle

process is straightforward since the gradient of the classifier is an unbiased driving term. Trainingfree guidance, however, is fundamentally more difficult. The primary challenge lies in leveraging a target predictor trained solely on clean samples to offer guidance on noisy samples. Although various approaches have been proposed [18, 63, 6, 2, 78] and are effective for some individual tasks, theoretical grounding and comprehensive benchmarks are still missing. Indeed, existing methods fail to produce satisfactory samples for label guidance even on simple datasets such as CIFAR10 (Figure 1). Moreover, the lack of quantitative comparisons between these methods makes it difficult for practitioners to identify an appropriate algorithm for a new application scenario.

This paper proposes a novel and general algorithmic framework for (and also named as) Training Free Guidance (TFG). We show that existing approaches are special cases of the TFG as they correspond to particular hyper-parameter subspace in our unified space. In other words, TFG naturally simplifies and reduces the study of training-free guidance, as well as the comparisons between existing methods, into the analysis of hyper-parameter choices in our unified design space. Within our framework, we analyze the underlying theoretical motivation of each hyper-parameter and conduct comprehensive experiments to identify their influence. Our systematic study offers novel insights into the principles behind training-free guidance, allowing for a transparent and efficient survey of the problem.

Based on the framework, we propose a hyper-parameter searching strategy for general downstream tasks. We comprehensively benchmark TFG and existing algorithms across 16 tasks (ranging from images to molecules) and 40 targets. TFG achieves superior performance across all datasets, outperforming existing methods by 8.5% on average. In particular, it excels in generating userrequired samples in various scenarios, regardless of the complexity of targets and datasets.

In summary, we (1) propose TFG that unifies existing algorithms into a design space, (2) theoretically and empirically analyze the space to propose an effective space-searching strategy for general problems, and (3) benchmark all methods on numerous qualitatively different tasks to present the superiority of TFG and the guideline for future research in training-free conditional generation algorithms. This advancement demonstrates the efficacy of TFG and establishes a robust and comprehensive benchmark for future research in training-free conditional generation algorithms.

2 Background

Generative diffusion model. A generative diffusion model is a neural network that can be used to sample from an unconditional distribution p 0 ( x ) with the support on any continuous sample space X [21, 62, 64, 27]. For instance, X could be [ -1 , 1] d × d × 3 representing the RGB colors of d × d images [4, 22], or R 3 d representing the 3D coordinates of molecules with d atoms [24, 74, 73]. Given a data x 0 sampled from p 0 ( x ) , a time step t ∈ [ T ] ≜ { 1 , · · · , T } , a corresponding noisy datapoint is constructed as x t = √ ¯ α t x 0 + √ 1 -¯ α t ϵ where ϵ ∼ N ( 0 , I ) and { ¯ α t } T t =1 is a set of pre-defined monotonically decreasing parameters used to control the noise level. Following [21], we further define α t = ¯ α t / ¯ α t -1 for t > 1 and α 1 = ¯ α 1 . The diffusion model ϵ θ : X × [ T ] ↦→X parameterized by θ is trained to predict the noise ϵ that was added on x t with p.d.f p t ( x t ) = ∫ x 0 p 0 ( x 0 ) p t | 0 ( x t | x 0 )d x 0 2 . In theory, this corresponds to learning the score of p t ( x ) [65], i.e.,

For sampling, we start from x T ∼ N ( 0 , I ) and gradually sample x t -1 ∼ p t -1 | t ( x t -1 | x t ) . This conditional probability is not directly computable, and in practice, DDIM [62] samples x t -1 via

where { σ t } T t =1 are DDIM parameters, ϵ ∼ N ( 0 , I ) , and

is the predicted sample given x t . According to the Tweedie's formula [11, 51], x 0 | t equals to the conditional expectation E [ x 0 | x t ] under perfect optimization of ϵ θ in Equation (1). It has been theoretically established that the above sampling process results in x 0 ∼ p ( x ) under certain assumptions.

Target predictor. For a user required target c , we use a predictor f c ( x ) : X ↦→ R + ∪ { 0 } 3 to represent how well a sample x is aligned with the target (higher the better). Here f c ( x ) can be a conditional probability p 0 ( c | x ) for a label c [62, 14], a Boltzmann distribution exp -e c ( x ) for any pre-defined energy function e c [31, 63, 38], the similarity of two features [47], or even their combinations. The goal is to samples from the conditional distribution

Training-based guidance for diffusion models. [66] proposes to train a time-dependent classifier to fit f c ( x t , t ) ≜ E x 0 ∼ p 0 | t ( ·| x t ) f c ( x 0 ) . This can be regarded as a predictor over noisy samples. Since

if we denote the trained classifier as f ( x t ) (that implicitly depends on c and model parameters), we can replace ϵ θ ( x t , t ) in Equation (3) by ϵ θ ( x t , t ) - √ 1 -¯ α t ∇ x t log f ( x t ) upon sampling to obtain unbiased sample x 0 ∼ p 0 ( x 0 | c ) . On the other hand, [23] proposes the classifier-free diffusion guidance approach. Instead of training a time-dependent predictor f , it encodes conditions c directly into the diffusion model as ϵ θ ( x , c, t ) and trains this condition-aware diffusion model with samplecondition pairs. Both methods have been proven effective when training resources are available.

This paper in contrast focuses on conditional generation in a training-free manner: given a diffusion model ϵ θ ( x , t ) and an off-the-shelf target predictor f ( x ) (we omit the subscript c below), we aim to generate samples from p 0 ( x | c ) without any additional training. Unlike training-based methods that can accurately estimate f ( x t , t ) , training-free guidance is significantly more difficult since it involves guiding a noisy data x t using f ( x ) defined over the clean data space.

2.1 Existing algorithms

Most existing methods take advantage of the predicted sample x 0 | t defined in Equation (3) and use the gradient of f ( x ) for guidance. We review and summarize five existing approaches below, and provide a schematic and a copy of pseudo-code in Appendix B for the sake of reference. Due to the variety in underlying intuitions and implementations, coupled with a lack of quantitative comparisons among these methods, it is challenging to discern which operations are crucial and which are superfluous, a problem we address in Section 3.

DPS [6] was initially proposed to solve general noisy inverse problems for image generation: for a given condition y and a transformation operator A , we aim to generate image x such that ∥A ( x ) -y ∥ 2 is small. For instance, in super-resolution task [71], the operator A is a down-sampling operator, and y is a low-resolution image. DPS replaces ∇ log f ( x t , t ) in Equation (5) by ∇ x t log f ( m ( x t )) . As suggested in [63], this corresponds to a point estimation of the conditional density p 0 | t ( x 0 | x t ) .

LGD [63] replaces the point estimation in DPS and proposes to estimate f ( x t , t ) with a Gaussian kernel E x ∼N ( x 0 | t ,σ 2 t I ) f ( x , t ) , where the expectation is computed using Monte-Carlo sampling [59].

FreeDoM [78] generalizes DPS by introducing a 'recurrent strategy' (called 'time-travel strategy' [39, 10, 70]) that iteratively denoises x t -1 from x t and adds noise to x t -1 to regenerate x t back and forth. This strategy empirically enhances the strength of the guidance at the cost of additional computation. FreeDoM also points out the importance of altering guidance strength at different time steps t , but a comprehensive study on which schedule is better is not provided.

MPGD [18] is proposed for manifold preserving tasks, e.g., the target predictor is supposed to generate samples on a given manifold. It computes the gradient of log f ( x 0 | t ) to x 0 | t instead of x t , i.e., ∇ x 0 | t log f ( x 0 | t ) to avoid the back-propagation through the diffusion model ϵ θ that is highly inefficient. This strategy is effective in manifold-preserving problems, but whether it can be generalized to general training-free problems is unclear. In addition to the computation difference, theoretical understanding on the difference between gradients to x 0 | t and x t is missing. 4

UGD [2] builds on FreeDoM, with the difference that it additionally solves a backward optimization problem ∆ 0 = arg max ∆ f ( x 0 | t +∆) and guides x 0 | t and x t simultaneously. UGD also implements the 'recurrent strategy' to further improve generation quality.

3 TFG: A Unified Framework for Training-free Guidance

Despite the array of algorithms available and their reported successes in various applications, we conduct a case study on CIFAR10 [30] to illustrate the challenging nature of training-free guidance and the insufficiency of existing methods. Specifically, for each of the ten labels, we use the pretrained diffusion model and classifiers from [7, 9] to generate 2048 samples, where the hyper-parameters are selected via a grid search for the fairness of comparison. We compute the FID and the label accuracy evaluated by another classifier [20] and present results in Figure 1. Even in such a relatively simple setting, all training-free approaches significantly underperform training-based guidance, with a significant portion of generated images being highly unnatural (when guidance is strong) or irrelevant to the label (when guidance is weak). These findings reveal the fundamental challenges and highlight the necessity of a comprehensive study. Unfortunately, comparisons and analyses of existing approaches are missing or primarily qualitative, limiting deeper investigation in this field.

3.1 Unification and extension

This sections introduces our unified framework for training-free guidance (TFG, Algorithm 1) and formally defines its design space in Definition 3.1. We demonstrate the advantage of TFG by drawing connections between TFG and other algorithms to show that existing algorithms are encompassed as special cases. Based on this, all comparisons and studies of training-free algorithms automatically become the study within the hyper-parameter space of our framework . This allows us to analyze the techniques theoretically and empirically, and choose an appropriate hyper-parameter for a specific downstream task efficiently and effectively, as shown in Section 4.

Algorithm 1 T rainingF ree G uidance

Definition 3.1. Given a denoising step T , the hyper-parameter space (design space) of Algorithm 1 is defined as

We use H TFG to represent the complete hyper-parameter space and H TFG ( N recur = N 0 ) to represent the subspace constrained on N recur = N 0 .

Definition 3.1 defines the hyper-parameter space spanned by TFG, where one hyper-parameter in H TFG is an instantiation of the framework. Intuitively, N recur controls the recurrence of the algorithm, N iter controls the iterating when computing ∆ 0 (Line 8), ¯ γ controls the extent we smooth the original guidance function f (Line 4), and ρ , µ control the strength of two types of guidance (Lines 7 and 8). A comprehensive explanation of the effect of each hyper-parameter can be found in Section 3.2.

Below is the major theorem showing that all algorithms presented in Section 2.1 correspond to special cases of TFG, thus unifying them into our framework and obviating the need for separate analyses.

Theorem 3.2. The hyper-parameter space of

MPGD [18] H MPGD is equivalent to H TFG ( N recur = N iter = 1 , ρ = 0 , ¯ γ = 0) .

LGD [63] H LGD is equivalent to H TFG ( N recur = 1 , N iter = 0 , µ = 0 ) .

UGD [2] H UGD is equivalent to H TFG (¯ γ = 0) .

DPS [6] H DPS is equivalent to H TFG ( N recur = 1 , N iter = 0 , µ = 0 , ¯ γ = 0) .

FreeDoM [78] H FreeDoM is equivalent to H TFG ( N iter = 0 , µ = 0 , ¯ γ = 0) .

The complete analysis and proof of Theorem 3.2 is postponed to Appendix C. It implies that existing algorithms are limited in expressivity, covering only a subset of H TFG. In contrast, TFG covers the entire space and is guaranteed to perform better. In addition, TFG streamlines nuances between existing methods, allowing for a unified way to compare and study different techniques. Consequently, the versatile framework that TFG provides can simplify its adaptation to various applications.

3.2 Algorithm and design space analysis

We now present a concrete analysis of TFG and its design space H in detail. Similar to standard classifier-based guidance, TFG guides x t at each denoising step t . To provide appropriate and informative guidance, TFG essentially leverages four techniques for guidance: Mean Guidance (Line 8) controlled by N iter , µ , Variance Guidance (Line 7) controlled by ρ , Implicit Dynamic (Line 4) controlled by ¯ γ , and Recurrence (Line 5) controlled by N recur .

Mean Guidance computes the gradient of ˜ f ( x ) to x 0 | t and is the most straightforward approach. However, this method can yield inaccurate guidance. To show this, notice that under perfect optimization we have x 0 | t = E [ x 0 | x t ] , and when p 0 ( E [ x 0 | x t ]) is close to zero, the predictor has rarely been trained on data from the region close to x 0 | t , making the gradient unstable and noisy. To mitigate this, one can iteratively add gradients of ˜ f ( x ) to x 0 | t , encouraging x 0 | t to escape low-probability regions.

Variance Guidance provides an alternative approach for improving the gradient estimation. The reason why we dub it variance guidance might be ambiguous, as the only difference is that the gradient is taken with respect to x t (Line 7) instead of x 0 | t (Line 8). The lemma below demonstrates that this essentially corresponds to a covariance re-scaled guidance.

Lemma 3.3. If the model is optimized perfectly, i.e., ϵ θ ( x , t ) = - √ 1 -¯ α t ∇ log p t ( x ) , we have where Σ 0 | t ≜ ∫ x p 0 | t ( x | x t )( x -E [ x 0 | x t ])( x -E [ x 0 | x t ]) ⊤ d x is the covariance of x 0 | x t .

Lemma 3.3 suggests that variance guidance refines mean guidance by incorporating the secondorder information of x 0 | x t , specifically considering the correlation among components within x 0 | t . Consequently, positively correlated components could have guidance mutually reinforced, while negatively correlated components could have guidance canceled. This also implies that mean guidance and variance guidance are intrinsically leveraging different orders of information for guidance. In TFG, variance guidance is controlled by ρ t .

Implicit Dynamic transforms the predictor f into its convolution via a Gaussian kernel N ( 0 , ¯ γ (1 -¯ α t ) I ) . This operation is initially introduced by LGD [63] to estimate p 0 | t ( x 0 | x t ) . However, it is unclear why the form is preselected as a Gaussian distribution. We argue that this technique is effective because it creates an implicit dynamic on x 0 | t . Specifically, starting from x 0 | t , it iteratively adds noise to x 0 | t , evaluates gradient, and moves x 0 | t based on the gradient. The repeating process converges to the density proportional to f ( x ) when N iter goes to infinity, driving x 0 | t to high-density regions. This explanation is justified by Table 1: the performance remains nearly un-

Table 1: Influence of the number of Monte-Carlo samples in estimating the expectation of Line 4. Both the FID and the accuracy remain unchanged when # Samples varies, suggesting that the number of samples is less important. More details are in Appendix E.1.

changed as we gradually decrease the number of Monte-Carlo samples in estimating the expectation (Line 4) down to 1 , implying that the preciseness of estimation is not essential, but adding noises is.

Recurrence helps strengthen the guidance by iterating the previous three techniques to obtain x t -1 and resample x t back and forth. This can be understood as an Ornstein-Uhlenbeck process [42] on x t -1 where Line 6 ∼ 9 corresponds to the drift term and x t -1 → x t (Line 10) the white noise term. Intuitively, it finds a trade-off between the error inherited from previous steps (the more you recur, the

increase

decrease

constant

N

recur

=

4

N

recur

=

2

N

recur

=

1

0.25

0.5

1.0

2.0

4.0

increase

decrease

constant

N

recur

=

4

N

recur

=

2

N

recur

=

1

0.25

0.5

1.0

2.0

4.0

FID

FID

CIFAR10

ImageNet

CIFAR10

ImageNet

increase

decrease

constant

N

iter

=

4

N

iter

=

2

N

iter

=

1

0.25

0.5

1.0

2.0

4.0

FID

increase

decrease

constant

N

iter

=

4

N

iter

=

2

N

iter

=

1

0.25

0.5

1.0

2.0

4.0

FID

less previous error stays) and the accumulated error in this step (the more you recur, the more error in the current guidance you suffer). Empirically, we also find that the generation quality improves and then deteriorates as we increase N recur .

4 Design Space of TFG: Analysis and Searching Strategy

Admittedly, a more extensive design space only yields a better performance if an effective and robust hyper-parameter searching strategy can be applied. For example, arbitrarily complex neural networks are guaranteed to have better optimal performance than simple linear models, but finding the correct model parameters is significantly more difficult. This section dives into this core problem by comprehensively analyzing the hyper-parameter space structure of H TFG, and further proposing a general searching algorithm applicable for any general downstream tasks.

The hyper-parameters of H TFG can be categorized into two parts: time-dependent vectors ρ , µ , and time-independent sacalars N recur , N iter , ¯ γ . While a grid search can potentially result in the best performance, performing such an extensive search in H TFG is highly impractical, especially considering the vector parameters ρ , µ . Fortunately, below we demonstrate that, if we decompose ρ into ¯ ρ · s ρ ( t ) (same for µ ) where ¯ ρ is a scalar and s ρ ( t ) is a 'structure' (a non-negative function) such that ∑ t s ρ ( t ) = T , then some structures are consistently better than others regardless of the other hyper-parameters. This allows us to pre-locate an appropriate structure for the given task and efficiently optimize the rest of the scalar hyper-parameters. Our analysis is conducted on the label guidance task on CIFAR-10 [30] and ImageNet [55], with experimental settings identical to Section 3.

Structure analysis. Motivated by the default structure selected in UGD and LGD, we consider three structures for both s ρ ( t ) and s µ ( t ) as

These structures are selected to be qualitatively different, while each is justified to be reasonable under certain conditions [18, 78, 2]. We leave the study of more structures to future works. The rest of the parameters are grid-searched for the comprehensiveness of the analysis. For s ρ ( t ) , we set N recur = { 1 , 2 , 4 } and ¯ ρ = { 0 . 25 , 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 } ; and for s µ ( t ) , we set N iter = { 1 , 2 , 4 } and ¯ µ = { 0 . 25 , 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 } . We run label guidance for each configuration and each of the ten labels on CIFAR10 (four labels on ImageNet, due to computation constraints).

As presented in Figure 2, the relationship between different structures remains unchanged when the rest of the parameters vary. For instance, on both datasets, the Validity-FID performance curves consistently move top-left (implying a better performance) when we switch from 'decrease' structure (red lines) to 'constant' structure (yellow lines) to 'increase' structure (blue lines) for both ρ , µ and different values of N recur and N iter. This invariant relationship is essential as it allows for an efficient hyper-parameters search in H TFG by first determining appropriate structures for s ρ ( t ) , s µ ( t ) under a simple subspace, and then selecting the rest scalar parameters.

Computation cost analysis. Among the scalars parameters, N recur and N iter directly influence the total computational cost, while ¯ ρ, ¯ µ, ¯ γ do not. With a certain range, performance increases when the value of N recur , N iter increase 5 , and the tradeoff between generation quality and computation time is presented in Figure 3: recurrence leads to a N recur times cost with clear performance gain; iteration (on x 0 | t ) results in less increase of computation time, and its effect plateaus. In practice, users can determine their values based on computation resources, but an upper bound of 4 suffices to unlock a near-optimal performance.

Time Cost (s / 256 samples)

N

recur

=

4

N

iter

=

4

N

iter

=

2

N

iter

=

1

N

recur

=

1

N

recur

=

2

N

recur

=

4

N

recur

=

2

N

recur

=

1

N

iter

=

4

N

iter

=

2

N

iter

=

1

0.25

0.5

1.0

2.0

4.0

FID

Searching strategy. The above analysis successfully simplify the task-specific hyper-parameter search problem without significant performance sacrifice. It remains to be decided the scalar values ¯ ρ, ¯ µ, ¯ γ . Here we propose a strategy based on beam search to effectively and efficiently select their values. Specifically, our searching strategy starts with an initial set T = { (¯ ρ init , ¯ µ init , ¯ γ init ) } , where these initial values are small enough to approximate TFG as an unconditional generation. At each searching step, for each tuple in T , we separately double the values of ¯ ρ , ¯ µ , and ¯ γ to generate up to 3 | T | new configurations. We conduct a small-sized generation trial for each new configuration and update T to be the top K configurations with the highest evaluation results that are determined by user requirements (e.g., accuracy, FID, or a combination). This iterative process is repeated until T stabilizes or the maximum number of search steps is reached. Notice that this process is conducted with a much smaller sample size, and consequently, the computation time is highly controllable.

5 Benchmarking

This section comprehensively benchmarks training-free guidance under the TFG framework and the design searching strategy in Section 4. We consider 7 datasets, 16 different tasks, and 40 individual targets with a total experimental cost of more than 2,000 A100 GPU hours. For comparison, we also run experiments for each of the existing methods (where the design searching is conducted in the corresponding subspace). All methods, tasks, search strategies, and evaluations are unified in our codebase, with details specified in Appendices D and E.

5.1 Settings

Diffusion models. (1) CIFAR10-DDPM [48] is a U-Net [54] model trained on CIFAR10 [30] images. (2) ImageNet-DDPM [7] is an larger U-Net model trained on ImageNet-1k [55] images. (3) Cat-DDPM is trained on Cat [12] images. (4) CelebA-DDPM is trained on CelebA-HQ dataset [26] that consists millions of human facial images. (5) Molecule-EDM [24] is an equivariant diffusion

model pretrained on molecule dataset QM9 [50] that performs molecule generation from scratch. (6) Stable-Diffusion ( v1.5 ) [53] is a latent text-to-image model that generate images with text prompts. (7) Audio-Diffusion 6 is a audio diffusion model based on DDPM trained to generate mel spectrograms of 256x256 corresponding to 5 seconds of audio.

Tasks. Our tasks (Table 2) cover a wide range of interests, including Gaussian deblur, super-resolution, label guidance, style transfer, molecule property guidance, audio declipping, audio inpainting, and guidance combination. Each task is run on multiple datasets or with multiple targets ( e.g. , different labels, molecular properties, styles).

Other settings. We consistently set the time step T = 100 and the DDIM parameter η = 1 . We consider N recur = 1 , N iter = 4 and use a single sample for Implicit Dynamic (Line 4) throughout all experiments and methods for fair comparison. For TFG, the structures of ρ and µ are set to 'increase' and the scalars ¯ ρ, ¯ µ, ¯ γ are determined via our searching strategy. We follow the setting in original papers if they specify their hyper-parameters. blueFor specific tricks in the code that are not mentioned in papers, we choose to align with original papers. Otherwise, values are determined via searching with 1 / 8 of the sample size and a maximum search step of 6 . For fairness of comparison, we use accuracy as the metric during the search and compare different algorithms on the metric, but we report both accuracy and FID.

5.2 Benchmarking results

We compare all six methods in Table 3. TFG outperforms existing algorithms in 13 over 14 settings, achieving an average guidance validity improvement of 7.4% compared to the best existing algorithm. Notice that we do not compare with the best algorithm in terms of generation fidelity because obtaining high realness samples is not our objective in training-free guidance, and an unconditional model suffices to generate high realness samples (with extremely low validity). Interestingly, different methods achieve the second best performance on different tasks, suggesting the variance of these methods, while TFG is consistent thanks to the unification.

We want to highlight that despite the superior performance of TFG, the key intention of our experiments is not restrained to comparing TFG with existing methods, but more importantly to systematically benchmark under the training-free guidance setting to see how much we have achieved in various tasks with different difficulties. Below we go through each task separately and conduct relevant ablation studies to provide a more fine-grained analysis.

Fine-grained label guidance. In addition to the standard label guidance, we for the first time study the out-of-distribution fine-grained label guidance under the training-free setting, a problem where no existing training-based methods are available. We consider the bird-species guidance using an

EfficienNet trained to classify 525 fine-grained bird species. This problem remains highly difficult for leading text-to-image generative models such as DALLE. Under recurrence, TFG can generate at most 2.24% of accurate birds, compared with the unconditional generation rate of 0 .

Recurrence on label guidance. We go back to the failure case we study in Section 3, i.e. , the standard label guidance problem on CIFAR10 where the training-based method offers an 85% accuracy, while the accuracy of TFG without recurrence accuracy is 52% only. As presented in Table 4, increasing N recur significantly closes the gap from 33% to 8%. Similar improvement is observed in other datasets as well.

Multiple guidance and bias mitigation. We next consider the scenario with multiple targets: control the generation of human faces based on gender and hair color (or age) using two predictors. It is well known that the label imbalance in CelebA-HQ causes classifiers to focus on spurious correlations [76] , such as using hair colors to classify gender, a biased feature we aim to avoid. The stratified performance of TFG on 'gender + age' and 'gender + hair' guidance are presented in Table 5. Despite the highly disparate performance, training-free guidance largely alleviates the imbalance: only 1% of images in CelebA are 'male + blonde hair', while the generated accuracy is 46.7%.

Molecule property guidance. To our knowledge, we are the first to study training-free guidance for molecule generation. We interestingly find in Table 3 that TFG is effective in guiding molecules towards desirable properties, yielding the highest guidance validity on 5 out of 6 targets with 5.64% MAE improvement over existing methods, verifying the generality of our approach as a unified framework in completely unseen domains. Notice that, unlike images, molecules with better validity usually have lower generation fidelity, a finding reflected in previous work [3].

Audio Guidance. We extend our investigation to the audio modality, where TFG achieves significant relative improvements over existing methods. Given that the audio domain is rarely explored in training-free guidance literature, our benchmarks will contribute to future research in this area.

6 Discussions and Limitations

Recently, training-free guidance for diffusion models has gained increasing attention and has been adopted in various applications. TFG is based on an extensive literature review over ten algorithmic papers for different purposes, including images, audio, molecules, and motions [34, 8, 43, 32, 19, 13, 16, 15, 45, 38, 68]. While we incorporate several key algorithms into our framework, we acknowledge that encompassing all approaches is impossible, as it would make the unification bloated and less practical. We seek to find a balance point by unifying most representative algorithms while keeping the techniques clear and easily studied.

An often discussed problem is why we care about training-free guidance, given the ever-growing community of language-based generative models such as the image generator of GPT4. In practice, there are countless conditional generation tasks where the conditions are hard to accurately convey to or represent by language encoders. For instance, it can fail to under a complex property of a molecule or generate CelebA-style faces. We give an illustrative analysis in Appendix A.1. Despite that training-free guidance is important, this paper does not systematically analyze what types of conditional generation are, in general, more suitable for the framework and what types are for language-based models. That said, training-free guidance is fundamentally difficult due to the misalignment between the training objective of target predictors and the diffusion, with a more detailed discussion in Appendix A.2. This paper does not comprehensively analyze this misalignment, and the gap between training-based and TFG remains high in some tasks like molecule property guidance. We hope that future works can analytically dive into these problems.

References

Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature , pages 1-3, 2024.

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 843-852, 2023.

Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations , 2023.

Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606 , 2021.

Mikołaj Bi´ nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401 , 2018.

Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis , 80:102479, 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Kieran Didi, Francisco Vargas, Simon V Mathis, Vincent Dutordoir, Emile Mathieu, Urszula J Komorowska, and Pietro Lio. A framework for conditional diffusion modelling with applications in motif scaffolding for protein design, 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International conference on machine learning , pages 8489-8510. PMLR, 2023.

Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106(496):1602-1614, 2011.

Jeremy Elson, John (JD) Douceur, Jon Howell, and Jared Saul. Asirra: A captcha that exploits interest-aligned manual image categorization. In Proceedings of 14th ACM Conference on Computer and Communications Security (CCS) . Association for Computing Machinery, Inc., October 2007.

Daniel Geng and Andrew Owens. Motion guidance: Diffusion-based image editing with differentiable motion estimators. arXiv preprint arXiv:2401.18085 , 2024.

Alexandros Graikos, Srikar Yellapragada, and Dimitris Samaras. Conditional generation from unconditional diffusion models using denoiser representations. arXiv preprint arXiv:2306.01900 , 2023.

Jiatao Gu, Qingzhe Gao, Shuangfei Zhai, Baoquan Chen, Lingjie Liu, and Josh Susskind. Control3diff: Learning controllable 3d diffusion models from single-view images. In 2024 International Conference on 3D Vision (3DV) , pages 685-696. IEEE, 2024.

Xu Han, Caihua Shan, Yifei Shen, Can Xu, Han Yang, Xiang Li, and Dongsheng Li. Trainingfree multi-objective diffusion model for 3d molecule generation. In The Twelfth International Conference on Learning Representations , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Representations , 2024.

Carlos Hernandez-Olivan, Koichi Saito, Naoki Murata, Chieh-Hsin Lai, Marco A MartínezRamirez, Wei-Hsiang Liao, and Yuki Mitsufuji. Vrdmg: Vocal restoration via diffusion posterior sampling with multiple guidance. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 596-600. IEEE, 2024.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research , 23(47):1-33, 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning , pages 8867-8887. PMLR, 2022.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 694-711. Springer, 2016.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems , 35:23593-23606, 2022.

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr \ 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 , 2018.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 , 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Jean-Marie Lemercier, Julius Richter, Simon Welker, Eloi Moliner, Vesa Välimäki, and Timo Gerkmann. Diffusion models for audio restoration. arXiv preprint arXiv:2402.09821 , 2024.

Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE conference on computer vision and pattern recognition , pages 1964-1971. IEEE, 2009.

Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. arXiv preprint arXiv:2311.00613 , 2023.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 , 2023.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning , pages 22825-22855. PMLR, 2023.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11461-11471, June 2022.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion-refinement paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.03530 , 2021.

Ross A Maller, Gernot Müller, and Alex Szimayer. Ornstein-uhlenbeck processes and extensions. Handbook of financial time series , pages 421-437, 2009.

Eloi Moliner, Jaakko Lehtinen, and Vesa Välimäki. Solving audio inverse problems with a diffusion model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Meinard Müller. Dynamic time warping. Information retrieval for music and motion , pages 69-84, 2007.

Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug-andplay conditional image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 20850-20860, 2023.

Kamal Nasrollahi and Thomas B Moeslund. Super-resolution: a comprehensive survey. Machine vision and applications , 25:1423-1468, 2014.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data , 1, 2014.

Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory , pages 388-394. Springer, 1992.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015.

Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.00855 , 2015.

Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. arXiv preprint arXiv:2102.09844 , 2021.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Alexander Shapiro. Monte carlo sampling methods. Handbooks in operations research and management science , 10:353-425, 2003.

Herbert A Simon. Spurious correlation: A causal interpretation. Journal of the American statistical Association , 49(267):467-479, 1954.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning , pages 32483-32498. PMLR, 2023.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems , 34:14151428, 2021.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Charles Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. In Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Probability Theory , volume 6, pages 583-603. University of California Press, 1972.

Haoyuan Sun, Bo Xia, Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigm of text-to-image generation with preferences through f -divergence minimization. arXiv preprint arXiv:2409.09774 , 2024.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention, 2021.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations , 2023.

Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence , 43(10):3365-3387, 2020.

Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature , 620(7976):1089-1100, 2023.

Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning , pages 38592-38610. PMLR, 2023.

Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923 , 2022.

Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. 2022.

Haotian Ye, James Zou, and Linjun Zhang. Freeze then train: Towards provable representation learning under spurious correlations and feature noise. In International Conference on Artificial Intelligence and Statistics , pages 8968-8990. PMLR, 2023.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2(3):5, 2022.

Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Trainingfree energy-guided conditional diffusion model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for finegrained category detection. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 , pages 834-849. Springer, 2014.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

A The Motivation of Studying Training-free Guidance

In this section, we argue that training-free guidance using off-the-shelf models is a crucial and timely research problem deserving more attention and effort. We begin by providing an illustrative analysis that highlights the limitations of current strong text-to-image generative models, underscoring the necessity for training-free guidance. Furthermore, we assert that training-free guidance remains a significant challenge, with previous literature underestimating its complexity. Given its necessity and inherent difficulties, we call for increased focus from the research community on this problem and offer our benchmarks and code base to help accelerate progress in this area.

A.1 Failure case of image generation with GPT4

口

ChatGPT 4>

口

Hello ChatGPT,can you create a figure of amolecule withPolarizability = 3?

externalelectricfield.Thenumericvalueof"3"you've mentioned needsa unit and morecontexttounderstandwhichmolecule

or type of molecule it refers to. Commonly, polarizability is expressed in units like A3 (cubic Angstroms) or other volume units.

Couldyouspecifywhich moleculeyou arereferring to,orprovide additional detailsabout itsstructure or composition?This

1.Anymoleculewithmore than5atomsisfine;2.Iwould likea standard3Dmolecule

visualization.

Got it!Foramoleculewithmorethanfive atoms andconsidering areasonablelevelofpolarizability,I'll createa figureof a

molecule likebenzene.Benzene hassixcarbonatoms and sixhydrogenatoms,forming aring structure,and it exhibitsnotable

electronicproperties.

I'llgenerateastandard3Dvisualizationofthebenzenemoleculeforyou.Let'sproceedwiththis.

Here is a 3Dmolecularmodel of benzene,whichfeatures a hexagonal ring structure.Thisvisualizationshows the arrangement of

It's hard for GPT4 to understand targets. In Figure 4, we ask GPT-4 to generate a molecule with polarizability α = 3 , which is a task we use to evaluate training-free guidance (refer to Figure 16 for visualization). We found that the GPT-4 generated molecule is apparently invalid and unrealistic: the generated molecule contains many carbon atoms with more than 4 bonds (the maximum allowed number is 4); and the generated molecule is apparently not a benzene which is claimed by the text outputs. From this case we may understand that it is hard to follow diverse user-defined instructions for the foundational generative models, where the user-defined targets may be subtle, fine-grained, combinatorial, and open-ended.

To this end, training-free guidance offers two key advantages: (1) It allows for greater specificity in target requirements by enabling the use of a differentiable objective function, making the generation process more steerable; (2) The objective function is plug-and-play, facilitating the addition of new targets and tasks to a pre-trained generative model. Since there is no need to retrain the diffusion or prediction models, this approach makes the generative process lightweight and applicable to various downstream tasks.

It's hard for GPT4 to capture the targeted distribution. Another important metrit for trainingfree guidance is the flexibility of choosing diffusion models. For the same target, we can switch from different diffusion models to change the unconditional background distribution. For example, it is hard for GPT4 to generate CelebA-like samples though it 'knows CelebA dataset very well':

口

ChatGPT4>

口

HelloChatGPT,doyouknowaboutCelebAdataset?

Yes, I'm familiar with the CelebA dataset!CelebA(CelebFaces Attributes Dataset) is a large-

scale face attributes dataset designed for research in the field of facial attribute recognition,

facedetection,and landmark(orfacialpart)localization.Itincludesover2oo,oo0imagesof

celebrities,each annotated with 40 attribute labels（like"Smiling"“Young","Wearing

Glasses", etc.) and 5 landmark locations (eyes,nose,mouth corners).The dataset hasbeen

widelyused inmachinelearning andcomputervisionresearch,particularlyfortasks

involvingfacialrecognitionandmodification.

ResearchersuseCelebAtodevelopandtestalgorithmsthatcanlearntoidentifyand

manipulatetheseattributesinunseenimages,advancingthecapabilitiesoffacialrecognition

it apopular choice for academic studies and experiments in image processing and machine

learning.

Can you generate an image of face of a young man in

celebA style?

Here isthegenerated imageofayoungman inthestyleof theCelebAdataset.

0

The flexibility to use different diffusion models provides an opportunity to generate a wider range of user-defined targets. With training-free guidance, individuals can select their preferred diffusion model to establish the background distribution and use the prediction model to steer the generation towards specific properties. This approach may represent a future direction for human-AI interaction.

DPS

DPS (fake)

LGD

LGD (fake)

0.8-

MPGD

MPGD (fake)

+

UGD-1

+

UGD-1(fake)

UGD-2

UGD-2(fake)

?

UGD-4

?

UGD-4(fake)

0.7

FreeDoM-1

FreeDoM-1(fake)

FreeDoM-2

★

FreeDoM-2(fake)

FreeDoM-4FreeDoM-4(fake)

training-based

Validity

0.6

Training-based (FID:25)

MPGD-fake(FID:28)

MPGD (FID:53)

0.5

Training

based

01=1S1=l07=S7=0=S=↓0+=1S+=↓6t=↓

t=5

t=0

0.4-

MPGD

(fake)

01=↓S1=407=4S=0=S=40+=S+=6+=↓

t=5

t=0

0.3

40

60

80

100

120

140

160

MPGD

FID

S=10I=S1=07=S7=↓0=↓S=0+=↓S=↓

t=0

A.2 The fundamental challenge of training-free guidance

Despite the array of algorithms available and their reported successes in various applications, we conduct a case study on CIFAR10 [30] to illustrate the challenging nature of training-free guidance and the insufficiency of existing methods. Specifically, we compare the training-based approach and training-free approach for the label guidance task on CIFAR10, with the diffusion model pretrained by [7], the training-based time-dependent classifier f ( x , t ) by [7], and the training-free standard label classifier f ( x ) pretrained only on clean CIFAR10 by [9]. and a 'fake' training-free classifier defined as f ( x , t ) | t =0 . The first serves as the oracle benchmark, while the second corresponds to the standard training-free guidance. The third setting, as considered in LGD [63], uses a 'fake' training-free classifier since its parameters are shared across different time steps t during training, resulting in an implicit regularization that is not available for practical predictors. This setting serves as a comparison to help identify the difficulty of training-free guidance.

Quantitative and qualitative results are shown in Figure 6. All training-free approaches significantly under-perform training-based guidance, with a significant portion of generated images being highly unnatural (when guidance is strong) or irrelevant to the label (when guidance is weak). A more clear illustration on this can be found in Figure 7. In terms of 'fake' classifiers, it leads to a remarkable difference from real training-free classifiers even under identical experimental settings. It generates

Training-based (FID=44.8,Acc=82%)

FreeDoM-4(FID=74.3,Acc=62%)

TFG-4(FID=65.5,Acc=88%)

much less messy images due to the implicit regularization from the training process, where noisy images are also 'seen' (although t is fixed to 0 upon guidance). Unfortunately, such types of predictors are inaccessible in practice (otherwise, we can use classifier-based guidance directly). From the comparison, we know that the key challenge of training-free guidance is the lack of a 'smoothing' classifier that can produce faithful guidance in the 'unseen' noisy image space.

The observation largely uncovers the essential difficulties of training-free guidance, and motivates us to systematically study techniques that can improve generation quality. Unfortunately, comparisons between existing techniques are ambiguous since different methods are tested on distinct and primarily qualitative applications, which in turn hinders the in-depth study in this field. To this end, we resolve to revisit this complicated scenario and design a clear and comprehensive framework for training-free guidance.

B Pseudo-code and schematics

We have presented the pseudo-code of TFG in Algorithm 1. Below, we provide a copy of the DPS (Algorithm 2), MPGD (Algorithm 3), FreeDoM (Algorithm 4), UGD (Algorithm 5), and LGD (Algorithm 6). Notice that LGD does not provide a pseudo-code, and we present their algorithm following their paper as a modification of DPS. We do not change the original algorithms' notations for reference. Please see the proof in Appendix C for the equivalence analysis. We provide a schematic of existing algorithms in Figure 8.

Algorithm 2 DPS - Gaussian

Algorithm 3 MPGD for pixel diffusion models

Algorithm 4 FreeDoM + Efficient Time-Travel Strategy

Algorithm 5 Universal Guidance (its α is our ¯ α )

Parameter: Recurrent steps k , gradient steps m for backward guidance and guidance strength s ( t ) ,

Required: z T sampled from N (0 , I ) , diffusion model ϵ θ , noise scales { α t } T t =1 , guidance function f , loss function ℓ , and prompt c

Calculate ∆ z 0 by minimizing ℓ ( c, f (ˆ z 0 +∆)) . with m steps of gradient descent Perform backward universal guidance by ˆ ϵ θ ← ˆ ϵ θ -√ α t / (1 -α t )∆ z 0

end if

end for

Algorithm 6 LGD (from DPS)

/gid00087

/gid00053

/gid00015

/gid00015

/gid00015

/gid00087

/gid00017

/gid00091

/gid00053

/gid00014

/gid00018

/gid00087

/gid00017

/gid00091

/gid00083

/gid00012

/gid00018

/gid00087

/gid00083

/gid00087

/gid00017 /gid00091 /gid00083

/gid00087

/gid00083/gid00014/gid00018

/gid00015

/gid00015

/gid00015

/gid00087

/gid00017 /gid00091

/gid00083

/gid00014 /gid00018

/gid00087

/gid00017 /gid00091 /gid00018

/gid00087

/gid00017

the

/gid00083

-th step

(a)

(b)

iterate

/gid00337

∇

/gid00087

/gid00083

/gid00001

/gid00087

/gid00017 /gid00091 /gid00083

)

/gid00337

∇

/gid00087

/gid00083

/gid00001

/gid00087

/gid00017 /gid00091 /gid00083

)

/gid00087

/gid00083

/gid00087

/gid00083/gid00014/gid00018

/gid00087

/gid00083/gid00014/gid00018

DPS

Diffusion

sampling

process

/gid00337

∇

/gid00087

/gid00083

/gid00001

/gid00087

/gid00017 /gid00091 /gid00083

)

/gid00087

/gid00083

/gid00087

/gid00083/gid00014/gid00018

/gid00087

/gid00083/gid00014/gid00018

UGD

LGD

/gid00087

/gid00083

/gid00087

/gid00083/gid00014/gid00018

MPGD

/gid00087

/gid00083

/gid00087

/gid00083/gid00014/gid00018

/gid00087

/gid00083/gid00014/gid00018

FreeDoM

/gid00087

/gid00009 /gid00017 /gid00010

)

/gid00087

/gid00009 /gid00018 /gid00010

)

/gid00087

/gid00009 /gid00019 /gid00010

)

/gid00087

/gid00017 /gid00091 /gid00083

/gid00087

/gid00009 /gid00017 /gid00010

/gid00087

/gid00009 /gid00018 /gid00010

/gid00087

/gid00009 /gid00019 /gid00010

/gid00087

/gid00083/gid00014/gid00018

/gid00087

/gid00083/gid00014/gid00018

/gid00087

/gid00083

/gid00337

∇

/gid00087

/gid00083/gid00001

log

Σ

/gid00072

/gid00001

f(

/gid00087

/gid00009

/gid00072

/gid00010

)

iterate

/gid00337

∇

/gid00087

/gid00017/gid00091/gid00083/gid00001

/gid00087

/gid00017 /gid00091 /gid00083

)

/gid00087

/gid00017 /gid00091 /gid00083

/gid00087

/gid00017 /gid00091 /gid00083

/gid00087

/gid00017 /gid00091 /gid00083

/gid00087

/gid00017 /gid00091 /gid00083

/gid00087

/gid00017 /gid00091 /gid00083

/gid00337

∇

/gid00087

/gid00017/gid00091/gid00083/gid00001

/gid00337

∇

/gid00087

/gid00017/gid00091/gid00083/gid00001

C Proofs

We prove Theorem 3.2 and Lemma 3.3 below.

C.1 Proof of Theorem 3.2

Proof. For each algorithm, we prove the equivalence of design space separately below. Notice that when ¯ γ = 0 , ˜ f degrades back to f .

MPGD (Algorithm 3). Below we demonstrate that any hyper-parameter { c t } T t =1 in Algorithm 3 is equivalent to the TFG with f ( x 0 | t ) = exp {-L ( x 0 | t ; y ) } and hyper-parameter

To show this, notice that since both ρ and ¯ γ are zero, Line 4 and Line 7 take no effect. When using the identical sampling function (Line 9), TFG generates x t -1 using

which is exactly the formula used in MPGD.

DPS (Algorithm 2) and FreeDoM (Algorithm 4). We prove both algorithms together as DPS is a special case of FreeDoM (without recurrence). Specifically, any hyper-parameter { ρ t } T t =1 , time travel step r and distance function D ( c , · ) in Algorithm 4 is equivalent to TFG with f ( x 0 | t ) = exp {-D ( c , x 0 | t ) } and hyper-parameter

To show this, notice that both algorithms have the identical resampling step from x t -1 to x t , so it suffices to prove that the formula to generate x t -1 in each recurrent step is the same. For FreeDoM, we have

and the last line equals the combination of Line 7 and Line 7 in TFG.

LGD (Algorithm 6). Any hyper-parameter { ζ t } T t =1 , n in LGD is equivalent to TFG with f ( x ) = exp {-ℓ y ( x ) } , sample size n in Line 4, and hyper-parameter

Notice that ¯ α t in TFG equals 1 / (1 + σ 2 t ) in the DPS algorithm. With this, the equivalence is clear from the pseudo-code of both algorithms.

UGD (Algorithm 5). Any hyper-parameter k, m, s ( t ) in UGD is equivalent to TFG with f ( x ) = exp {-ℓ ( c, f ( x )) } and hyper-parameter

where δ t is the coefficient of ϵ θ ( x t , t ) in sampler S in the UGD algorithm (which is negative). To show this, notice that ∆ t = ρ t ∇ x t log ˜ f ( x 0 | t ) = -ρ t ∇ x t ℓ ( c, f ( x 0 | t )) = √ α t s ( t ) δ t ∇ x t ℓ ( c, f ( x 0 | t )) . By replacing this into Line 9, the equivalence of guidance Variance Guidance can be easily observed. A similar deduction can be made for the mean guidance as well.

C.2 Proof of Lemma 3.3

Proof. Recall that x 0 | t = x t - √ 1 -¯ α t ϵ θ ( x t ,t ) √ ¯ α t . According to a simple chain rule, we have

The perfect optimization assumption implies the relationship between ϵ θ and the score of p t ( x t ) , which we leverage and obtain

Thus, it remains to draw a connection between the conditional covariance between Σ 0 | t and ∇ 2 x t log p t ( x t ) . Omit the subscript x t in ∇ x t , we have

(15)

Notice that

Replacing the LHS in the above equation in Equation (14), and noticing that E [ x 0 | x t ] = x t +(1 -¯ α t ) ∇ log p t ( x t ) √ ¯ α t , we have

The proof is finished by substituting ∇ 2 log p t ( x t ) in Equation (12) by the above result.

D Task details

D.1 Gaussian Deblur

In computer vision, the Gaussian deblur task addresses the challenge of restoring clarity to images that have been blurred by a Gaussian process. Gaussian blur, a common image degradation, simulates effects such as out-of-focus photography or atmospheric disturbances. It is characterized by the convolution of an image with a Gaussian kernel, a process that spreads the pixel values outwards, leading to a smooth, uniform blur [33]. The deblurring task seeks to reverse this effect, aiming to retrieve the original sharp image.

Guidance target. Specifically, we apply a 61 × 61 sized Gaussian blur with kernal intensity 3 . 0 to original 256 × 256 images. A random noise with a variance of σ 2 = 0 . 05 2 is added to the noisy images. If we denote the above process as a blurring operator A blur : x → y , where x ∈ R 256 × 256 × 3 , y ∈ R 256 × 256 × 3 are original images and noisy images, then the target of Gaussian Deblur is to generate a clean image x 0 such that:

which means if we project the generated image into the noisy space, the noisy samples A blur ( x ) should be similar to the ground truth noisy images y .

Evaluation metrics. In our experiments, we evaluate each guidance method on a set of 256 samples generated by Cat-DDPM. We apply the FID [26] to assess the fidelity, Learned Perceptual Image Patch Similarity (LPIPS) [80] to evaluate the guidance validity.

LGD

UGD

TFG

D.2 Super Resolution

Super-resolution in computer vision refers to the process of enhancing the resolution of an imaging system, aimed at reconstructing a high-resolution image from one or more low-resolution observations. This technique is fundamental in overcoming the inherent limitations of imaging sensors and improving the detail and quality of digital images. Super-resolution has broad applications, ranging from satellite imaging and surveillance to medical imaging and consumer photography [46].

Guidance target. Specifically, we simply down-sample to original 256 × 256 images to the resolution of 64 × 64 . A random noise with a variance of σ 2 = 0 . 05 2 is also added to the noisy images. If we denote the above process as a degradation operator A down : x → y , where x ∈ R 256 × 256 × 3 , y ∈ R 256 × 256 × 3 are original images and down-sampled images, then the target of superis to generate a high resolution image x 0 such that:

which means if we project the generated image into the downsampled image space, the downsampled samples A down ( x ) should be similar to the ground truth downsampled images y .

Evaluation metrics. Similar to Gaussian Deblur, we evaluate each guidance method on a set of 256 samples generated by Cat-DDPM. We apply FID [26] to assess the fidelity, Learned Perceptual Image Patch Similarity (LPIPS) [80] to evaluate the guidance validity.

LGD

UGD

TFG

D.3 Label Guidance

Label guidance is a standard task for conditional generation studied in previous literature [7, 23]. The target is to generate images conditioned on a certain label. We found this standard task is rarely studied in training-free guidance work and there exist an evident performance gap between training-based guidance and existing training-free guidance as shown in Section 3.

Label sets. In our experiments, we studied labels from CIFAR10 [30] and ImageNet [55]. We average the results on 10 labels from CIFAR10 if there is no extra explanation. For ImageNet, which is resource-hungry to do comprehensive inference, we randomly select 4 labels (111, 222, 333, 444) to evaluate the methods. The corresponding label-ID and their names are as follows:

Guidance target. For each dataset, we use the output probability of a pre-trained classifier h ( · ) as the target probability. Our target is to maximize the certain classification probability of a given label, i.e. ,

where i is the label-ID of the target label, and h ( · ) is the logits of x 0 produced by the pre-trained classifier. For CIFAR10 and ImageNet, we use a pre-trained classifier based on ResNet [17] and VIT [9] that are provided from [75] and [9] respectively. The image resolution for CIFAR10 and ImageNet are 32 × 32 and 256 × 256 respectively.

Evaluation metrics. We follow the image generation literature to use Fréchet inception distance (FID) [20] to assess the fidelity of generated images. The reference images are filtered from the entire dataset of CIFAR10 or ImageNet with the target label, and we set sample sizes as 2560 and 256 for CIFAR and ImageNet respectively. For validity, we use another pre-trained classifier to compute accuracy other than the one used in providing guidance to avoid over-confidence:

For CIFAR10, we use a pre-trained ConvNeXT [36] downloaded from HuggingFace Hub 7 . And for ImageNet, we use the pre-trained DeiT [69] downloaded from HuggingFace Hub 8 .

DPS

MPGD

UGD-4

LGD

TFG-4

D.4 Combined Guidance

An interesting scenario for conditional generation is to assign multiple targets for a single sample. Conditional generation with multiple conditions is crucial in machine learning for enhancing the relevance and applicability of AI across complex, real-world scenarios. It enables models to produce more contextually appropriate and personalized outputs, crucial for fields requiring high customization.

Motivations. Combined guidance is to use multiple target functions to guide the same sample towards multiple targets for the same sample. It is more efficient for training-free guidance to do combined guidance as the space of combinatorial targets is potentially huge, which makes it unrealistic to train all target combinations for training-based guidance. We also find it related to the topic of spurious correlations [60], where certain combinations of attributes may dominant the other combinations. For example, hair color may have a strong correlation with gender in CelebA dataset [37]. It is beneficial to explore training-free guidance on reducing the bias of generation models trained on these biased data and address the concerns related to fairness and equality.

Guidance target. We study combined guidance on CelebA-DDPM, which is trained on CelebAHQ [26] dataset. The image resolution is 256 × 256 . We choose two settings of combined guidance with two attributes: (gender, hair color) and (gender, age). Each attribute has two labels, where gender ∈ { male , female } , age ∈ { young , old } , and hair color ∈ { black , blonde } . We have a binary classifier for each attribute that is downloaded from HuggingFace Hub 91011 . The target is to sample images that maximize the marginal probability:

where p target ( x 0 ) is computed using label guidance as shown in Appendix D.3.

Evaluation metrics. As it is hard to filter many reference images for combined targets in CelebAHQ dataset, we adopt Kernel Inception distance (KID) [5] to assess fidelity of generated samples using 1,000 random sampled images of CelebA-HQ as reference images. We generate 256 samples for each evaluated method. We follow MPGD [18] to use the logarithm of KID, i.e. KID (log). For validity, we use another three attribute classifiers to compute the accuracy considering the conjunction of attributes:

The evaluation classifiers are also downloaded from HuggingFace Hub 121314 .

DPS

LGD

MPGD

FreeDoM

UGD

TFG

D.5 Fine-grained Guidance

Fine-grained classification is a specialized task in computer vision that focuses on distinguishing between highly similar subcategories within a larger, general category. This task is particularly challenging due to the subtle differences among the objects or entities being classified. For example, in the context of animal species recognition, fine-grained classification would involve not just distinguishing a bird from a cat, but identifying the specific species of birds, such as differentiating between a crow and a raven [79].

Studying fine-grained generation in the context of generative models like Stable Diffusion or DALL-E presents unique challenges due to the inherent complexity of generating highly detailed and specific images. Fine-grained generation involves creating images that not only belong to a broad category but also capture the subtle nuances and specific characteristics of a narrowly defined subcategory. For example, generating images of specific dog breeds in distinct poses or environments requires the model to understand and replicate minute details that distinguish one breed from another.

Motivations. To develop personalized AI, it is important to explore if the foundational generative models can synthesize fine-grained, accurate target samples according to user-defined target. However, this usually requires high-quality and detailed training data, and the model should be highly sensitive to small variations in input to accurately produce the desired output, which can be difficult for strong text2image generation models DALL-E 15 or Imagen 16 . We first study this problem in a training-free guidance scenario.

Guidance target. Westudy the out-of-distribution fine-grained label guidance on ImageNet-DDPM, which learns the generation of some species of birds but not comprehensively. We use an EfficientNet trained to classify 525 fine-grained bird species downloaded from HuggingFace Hub. 17 The classifier is trained on Bird Species dataset on Kaggle 18 . Wefollow the same way in label-guidance to maximize softmax probability for target fine-grianed label. We randomly sample 4 labels (111, 222, 333, 444) in Bird Species dataset, which are:

Evaluation metrics. Similar to label guidance, we use FID to evaluate generation fidelity by filtering data of the target label as reference images. We also compute the FID with 256 sampled images. For accuracy, we adopt another downloaded pre-trained classifier trained on Bird Species dataset from HuggingFace Hub. 19 .

D.6 Style Transfer

Style transfer is a significant task in computer vision (CV) that focuses on applying the stylistic elements of one image onto another while preserving the content of the target image. This task is pivotal because it bridges the gap between artistic expression and technological innovation, allowing for the creation of novel and aesthetically pleasing visual content. Applications of style transfer are vast, ranging from enhancing user engagement in digital media and advertising to aiding in architectural design by visualizing changes in real-time.

Guidance target. The target in our experiments is to guide a text-to-image latent diffusion model Stable-Diffusion-v-1-5 [53] 20 to generate images that fit both the text input prompts and the style of the reference images. The guidance objective involves calculating the Gram matrices [25] of the intermediate layers of the CLIP image encoder for both the generated images and the reference style images. The Frobenius norm of these matrices serves as the metric for the objective function. Specifically, for a reference style image x ref and a decoded image D ( z 0 | t ) generated from the

estimated clean latent variable z 0 | t , we compute the Gram matrices G ( x ref ) and G ( D ( z 0 | t )) . These matrices are derived from the features of the 3rd layer of the image encoder, in accordance with the methodologies described in MPGD [18] and FreeDoM [78]. The target function is computed as follows:

where ∥ · ∥ 2 F denotes the Frobenius norm of a matrix.

Evaluation metrics. We use Style score and CLIP score to assess the guidance validity and fidelity, respectively. For reference style images and text prompts, we select 4 images from WikiArt [56] that are also used by MPGD [18], and 64 text prompts from Partiprompts [77]. For each style, we generate 64 images based on the 64 different text prompts. To avoid over-confidence of CLIP score, we use two different CLIP models downloaded from HuggingFace Hub to compute guidance and evaluation metrics, respectively. 2122 . The style images and examplar prompts are shown as follows.

山

DPS

LGD

MPGD

FreeDoM

UGD

TFG

D.7 Molecule Property Guidance

Setup. Our benchmark setup generally follows [24, 3] but with certain specifications to guarantee the overall framework abides by the training-free regime. For dataset, we employ QM9 [50] and adopt the split in [24] with 100,000 training samples. Following [24] and [3], the training set is further split into two halves that guarantees there is no data leakage. The first half is leveraged to train a property prediction network with EGNN [57] as the backbone, which serves as the ground truth oracle to provide the label used for MAE computation. We reuse the checkpoints released by [3] for the labeling network for all 6 properties. The second half is used to train the diffusion model as well as the guidance network. We adopt the unconditional generation version of EDM [24] as the diffusion model. The guidance network in general takes the same architecture as defined by [3] that, again, features EGNN as the backbone but outputs a single scalar as the predicted quantum mechanics property. The only difference lies in at training time we mask the diffusion time step by zeros and always use the original clean molecule structure as input, ensuring training-free objective. All the pretrained models are trained separately for different properties. At sampling time, we employ DDIM [62] sampler with 100 sampling steps, as opposed to [24, 3] that take 1000 sampling steps.

Guidance target. We study training-free guided generation of molecules on 6 quantum mechanics properties, including polarizability ( α ), dipole moment ( µ ) , heat capacity ( C v ), highest orbital energy ( ϵ HOMO), lowest orbital energy ( ϵ LUMO) and their gap ( ∆ ϵ ). Denote the oracle property prediction network as E . Our guidance target in this case is given by

where x is the input molecule and c is the target property value.

Evaluation metrics. We use Mean Absolute Error (MAE) and validity as our evaluation metrics. In particular, MAE is computed between the target value and the predicted value gathered from the labeling network. Validity is computed by RDKit which measures whether the molecule is chemically feasible. We generate 4096 molecules for each property for evaluation.

40

60

80

100

TFG

(Ours)

DPS

LGD

MPGD

FreeDoM

UGD

Polarizability

(𝛼)

D.8 Audio Declipping

Audio declipping is a task in digital audio restoration where distorted audio signals are repaired. Clipping occurs when the amplitude or frequency of an audio signal exceeds the maximum limit that the recording system can handle, leading to a harsh, distorted sound with portions of the waveform 'cut off.' Declipping aims to reconstruct the missing parts of these clipped waveforms, restoring the audio's original dynamics and reducing distortion. This process improves the quality and clarity of the audio, making it more pleasant to listen to while preserving the original sound's integrity.

Guidance target. Specifically, we apply a distortion operation to zero out the high frequency and low frequency (for the highest and lowest 40 dimensions) signal in the space of mel spectrograms. If we denote the above process as a blurring operator A blur : x → y , where x ∈ R 256 × 256 , y ∈ R 256 × 256 are mel spectrograms and noisy mel spectrograms for 5s audios, then the target of Audio Declipping is to generate a clean audio x 0 such that:

which means if we project the generate mel spectrogram into the noisy space, the noisy samples A blur ( x ) should be similar to the ground truth noisy mel spectrogram y .

Evaluation metrics. In our experiments, we evaluate each guidance method on a set of 256 samples generated by Audio-diffusion. We apply the Dynamic time warping (DTW) [44] to assess the guidance validity, and Fréchet Audio Distance (FAD) [28] to assess the generation fidelity.

D.9 Audio Inpainting

Audio inpainting is a digital audio restoration task that involves filling in missing or corrupted segments of an audio signal. Similar to image inpainting in the visual domain, this technique reconstructs the missing portions of sound by analyzing the surrounding context and seamlessly restoring the lost information. Applications of audio inpainting range from repairing damaged recordings to reconstructing gaps in audio streams due to data loss. The goal is to produce a natural-sounding result that preserves the continuity and overall quality of the original audio.

Guidance target. Specifically, we apply a distortion operation to zero out the middle 80 dimensions in the space of mel spectrograms. If we denote the above process as a blurring operator A blur : x → y , where x ∈ R 256 × 256 , y ∈ R 256 × 256 are mel spectrograms and noisy mel spectrograms for 5s audios, then the target of Audio Declipping is to generate a clean audio x 0 such that:

which means if we project the generate mel spectrogram into the noisy space, the noisy samples A blur ( x ) should be similar to the ground truth noisy mel spectrogram y .

Evaluation metrics. In our experiments, we evaluate each guidance method on a set of 256 samples generated by Audio-diffusion. We apply the Dynamic time warping (DTW) [44] to assess the guidance validity, and Fréchet Audio Distance (FAD) [28] to assess the generation fidelity.

E Experimental Details

E.1 Details of Table 1

In Table 1, we study the effect of Monte-Carlo sample sizes in estimating the expectation of Line 4 in TFG algorithm. As the noise is added on both Mean Guidance ( ∆ 0 ) and Variance Guidance ( ∆ t ), we decouple the effect into adding noise solely on ∆ 0 (Mean only) or ∆ t (Variance only). In the setting of Variance only, we set µ = 0 , N iter = 0 , N recur = 4 , s ρ ( t ) = 'increase', and pick the best ¯ ρ and ¯ γ via hyper-parameter search. In the setting of Mean only, we set ρ = 0 , N iter = 4 , N recur = 1 , s µ ( t ) = 'increase', and pick the best ¯ µ and ¯ γ via hyper-parameter search. We found that the sample size used in Monte-Carlo method play a neglect-able role on the performance if we set the optimal hyper-parameter. It is also noteworthy that the Monte-Carlo sampling does affect the performance of generated quality. For example, we can find that different targets shown in Appendix E.3 have different searched ¯ γ . This indicates that the best ¯ γ for many targets are apparently not zero.

E.2 Comparison with grid search

We compare the performance of our beam search parameters with the full grid search ones on CIFAR10 label guidance task (Table 10). Overall, the performance of both search methods is identical, while grid search is much slower than our search strategy, indicating that our beam search strategy is effective and efficient.

E.3 Detailed results of each target and hyper-parameters

In this section, we present the hyper-parameters searched via the strategy introduced in Section 4 and the corresponding experimental results for TFG as shown in Table 11. We list several observations below.

Overall, optimal parameters vary widely between problems and datasets. For example, even with the same model and objective (e.g., label classifier on ImageNet or CIFAR10), the best hyperparameters vary widely from target to target. This highlights the importance of hyperparameter search.

The improvement of TFG over existing methods depends heavily on the difference between the optimal parameters and the subspaces of existing methods. For example, the ¯ ρ for UGD is the same as TFG for gender-age guidance task, where TFG only has 0.133% validity improvement over UGD. On the contrary, their values differ on the fine-grained classification task, and TFG has an 18.7% validity improvement over UGD. Overall, we suppose this depends on whether the optimal parameter lies in the subspace that existing methods can find.

Though the baselines mentioned in our paper should be a special case of TFG, the results for the highest MO energy guidance in Table 3 show that MPGD outperforms TFG slightly. We want to point out that the reason TFG could occasionally have slightly worse performance in practice is due to the beam search computation limit we currently pose. More specifically, we allow TFG to search at most six steps (for all hyper-parameters) and all other methods for seven steps (in their subspaces). For the MO energy task, the searched parameter for MPGD is that ¯ µ = 0 . 016 (this is the only parameter that we need to search for MPGD), where the best (and last step) of TFG is that ¯ µ = 0 . 004 (because it uses one step to double another parameter). If we allocate more computational budget for the beam search steps, TFG will outperform MPGD on this target (in fact, eight steps suffice).

E.4 Tricks implemented in FreeDoM codebase

In the codebase of FreeDoM 23 , the schedule of guidance strength for different applications is different. For example, the guidance strength has a schedule coefficient √ ¯ α t for face generation, and the schedule for style transfer is complex and involves a correction term, the mean of gradients' norm, and a specific constant coefficient 0 . 2 . The paper does not mention this particular schedule, leaving

the rationale for choosing these schedules unclear. We choose not to include the tricks and find that with our unified hyper-parameter searching strategy, the performance of FreeDoM is similar.

E.5 Hardware and Software

We run most of the experiments on clusters using NVIDIA A100s. We implemented our experiments using PyTorch [49] and the HuggingFace library. 24 Overall, we estimated that a total of 2,000 GPU hours were consumed.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have both theoretically and empirically justified our contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Sec. C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Sec. D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See supplementary file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Sec. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiment is conducted on an extensively large scale, and existing works of the same line were not reported since the numbers are stable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Sec. 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: It is not applicable to the concern of this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the related papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not include new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve human subjects and studies.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Training-Free Guidance (TFG), a novel framework designed to enhance the generation of samples with desired properties using diffusion models, without necessitating additional model training. TFG aims to resolve the shortcomings of existing training-free methods by offering a unified algorithmic framework that simplifies the comparison and application of such methods across a wide range of tasks. By theoretically and empirically analyzing a hyper-parameter design space within this framework, the authors develop an effective strategy for hyper-parameter selection applicable to various tasks. Their comprehensive benchmarks across multiple tasks and targets demonstrate TFG's superior performance, achieving an average improvement of 7.4% over existing methods.\n\nSTRENGTHS:\n1. Given the burgeoning interest in the area of training-free guidance, the establishment of a unified benchmark as presented in this paper is a commendable contribution that holds the potential to significantly advance research in this field. The authors' efforts in this direction are highly appreciated and underscore the importance of standardized benchmarks for facilitating future developments.\n\n2. The experiments conducted in this study are extensive, reflecting a high degree of diligence and thoroughness. Such a comprehensive experimental approach is commendable and warrants recognition. Consequently, I believe this aspect of the paper merits a positive evaluation for its contribution to validating the proposed framework and its applicability across a diverse range of tasks.\n\nWEAKNESSES:\nThere are several areas where improvements could significantly enhance its contributions. Addressing these points satisfactorily would make a strong case for elevating the paper's status to \"Accept\" or \"Strong Accept\".\n\n1. While the paper provides valuable insights into a specific line of training-free guidance, it is important to acknowledge the existence of a broader spectrum of works outside this category. This observation suggests that the title of the paper may slightly overreach, potentially implying a more comprehensive coverage than is actually presented. (Question 1)\n\n2. The paper commendably supports the motivations behind conditional diffusion in Appendix A, offering a solid foundation for its relevance. However, the rationale for focusing specifically on training-free approaches appears less extensively articulated. (Question 2)\n\n3. The authors claimed that they have theoretically grounded their unified framework. However, upon review, the theoretical underpinnings presented seem to require further elaboration to fully substantiate this claim. (Question 3, 4, 5)\n\n4. The authors claimed that \"the studies of training-free methods become the study within the hyperparameter space of our framework\". Nonetheless, it appears that the search space defined within this framework might not fully encompass the range of hyper-parameters considered in existing literature. This limitation could potentially restrict the framework's applicability or comparative analysis capabilities. (Question 6, 7)\n\nQUESTIONS:\n1. The field of training-free guidance encompasses a wide array of studies beyond those specifically addressed in this paper (e.g. [1]-[3]). Could you elaborate on how these works relate to the scope and contributions of your paper?\n\n2. Your paper operates under the assumption that the forward model is known, leveraging training-free guidance for solving inverse problems. Given that knowing the forward model allows for the generation of a large number of samples to train a conditional diffusion model at a relatively low computational cost (e.g., 10 A100 GPU hours) and training-based approaches are much better than training-free ones, could you discuss the significant motivations of adopting a training-free approach in this context?\n\n3. The theoretical foundation of your paper seems to rest significantly on Lemma 4.1, which revisits the variance of MMSE estimator of the signal corrupted by Gaussian noise (e.g., (2.8) in [4]). This formula is widely adopted in diffusion papers (e.g., [5]). Given its established nature and previous applications, could you elaborate on how this lemma specifically contributes to the novel aspects of your framework?\n\n4. Concerning Lemma 4.1, it appears there is no assurance that the generated image accurately follows the conditional distribution, nor is there a guarantee that the loss decreases at each iteration as suggested by equation (7). Could you provide further clarification or additional theoretical support to address these concerns?\n\n5. Several techniques introduced in the paper lack direct theoretical underpinning:\n- The concept of \"time-travel\" being an Ornstein-Uhlenbeck process is intriguing but lacks a detailed derivation. Could you expand on how the theory of the Ornstein-Uhlenbeck process quantifies the benefits of time-travel in your framework?\n- The selection of hyperparameters seems not to be grounded in theory. Could you discuss the rationale behind these choices and any potential theoretical support?\n\n6. The paper posits that the study of training-free methods can be encapsulated within the hyperparameter space of your framework. However, specific hyperparameter settings critical for the performance in existing works, such as the step sizes used in Face generation (at.sqrt()) and Style transfer ((correction * correction).mean().sqrt().item() * unconditional_guidance_scale / (norm_grad * norm_grad).mean().sqrt().item() * 0.2) in FreeDoM, are not explicitly covered. Could you address the omission of these settings and their impact on the comprehensiveness of your study?\n\n7. The hyperparameter search settings for the baselines in your comparative analysis are not disclosed, raising questions about the fairness and validity of the comparisons. Could you provide more details on these settings to ensure a transparent and equitable comparison?\n\n[1] Feng, Weixi, et al. \"Training-free structured diffusion guidance for compositional text-to-image synthesis.\" ICLR 2023.\n\n[2] Chen, Minghao, et al. \"Training-free layout control with cross-attention guidance.\" WACV 2024.\n\n[3] Mo, Sicheng, et al. \"Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition.\" CVPR 2024.\n\n[4] Efron, Bradley. \"Tweedie’s formula and selection bias.\" JASA 2011.\n\n[5] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representation.\" ICLR 2024.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a framework (TFG) for training-free guidance of unconditional diffusion models, enabling their application to conditional generation tasks such as super-resolution, deblurring, etc. via the use of a predictor that evaluates the quality of a clean sample.  TFG, as in related past methods, aim to sample from the conditional distribution of samples defined in Equation 4 without either training a conditional diffusion model or a predictor evaluating the quality of noisy samples as in classifier guidance.  The gradient of the predictor on noisy samples combined with the unconditional score produce the correct conditional score function in Equation 5.  The challenge of training-free methods is to approximate this gradient somehow and a variety of approaches have been previously proposed.  TFG defined in Algorithm 1 is shown to include five such previous training-free methods as special cases for particular hyperparameters, demonstrating that these methods can be viewed and understood in a unified framework.  The authors then propose a procedure to optimize TFG's hyperparameters jointly.  By leveraging the larger design space with optimized hyperparameters, performance gains are observed across 14 tasks, 6 diffusion models, and 38 target predictors compared to the past methods subsumed by TFG.\n\nSTRENGTHS:\n- The TFG framework appears to be a non-trivial, novel incorporation of recent training-free methods and helps contextualize the relationships of these methods to one another.  Training-free guidance of diffusion models is an unsolved problem with many recent papers and placing a portion of this literature into a unified framework is a valuable contribution.  \n- The framework also led to substantive performance gains on generation validity in the comprehensive benchmarks.  The benchmark evaluations included a good variety of tasks, models, and predictors, including particularly out-of-distribution fine-grained label guidance and molecule property guidance.\n- The paper is generally well-written and organized.\n\nWEAKNESSES:\n- Some statements and claims are overly broad.  The authors claim all existing training-free approaches fit into their framework which is unlikely.  Many training-free methods have been proposed and the authors could better place their framework in the context of recent literature.  Section 3 Figure 1's demonstration that training-based conditional generation outperforms training-free is unsurprising, given that training-free methods are approximations.\n- While hyperparameter search was helpful, the unified design space discussion in Section 4.1 led to limited theoretical insight.  The authors could consider expanding this section to improve their methodological contributions.\n- The hyperparameter setting strategy described in Section 4.2, key to getting gains versus past methods, could use more detailed explanation (see questions 2-4 below). \n- The benchmark Section 5 decides to not compare generation fidelity in favor of comparing the best algorithm in terms of generation validity.  However, throughout the rest of the paper (e.g. figure 1, 2, and 3) tradeoffs between fidelity and validity are emphasized.  Looking at Table 2, TFG is sometimes but not always the best algorithm in terms of both fidelity and validity simultaneously.  The focus on only validity seems unjustified here.\n\nQUESTIONS:\n1. Why does increasing $N_{recur}$ and $N_{tier}$ eventually hurt performance?\n2. In section 4.2, does the \"structure analysis\" that \"increase\" is best hold more generally on other tasks than label guidance?   \n3. The number of Monte Carlo samples in the Implicit Dynamic is set to 1 based on Table 1, but that choice needs more justification.  Does more samples than 4 help?  Why is implicit dynamic helpful with few samples?  This seems contrary to the original motivation of estimating an average.\n4. In section 4.2, the beam search in \"Searching strategy\" could use more detail.  What sample sizes are used and how did that enable quick search? Also, what metric is used when deciding the top K candidates in this search?\n\nLIMITATIONS:\nLimitations are briefly discussed in Section 6, focusing on why training-free guidance remains relevant given language-based image generators.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper scrutinizes existing works on training-free guidance in diffusion models and proposes a unified framework that includes all existing methods as special cases. With this unified framework, this work presents a detailed and informed investigation of the design choices and hyperparameters within this framework. Additionally, it proposes a comprehensive benchmark involving 14 task types and 38 targets to evaluate the performance of this unified framework by optimizing the hyperparameters in the design choices.\n\nSTRENGTHS:\n1. The proposed unified framework is a very interesting and novel summarization of the methods in existing works. Existing training-free guidance often uses different notations and different ways to formulate the problem. It is very helpful that this framework elucidates the design choices in training-free guidance.\n\n2. The proposed benchmark is very comprehensive, and I believe it will certainly be helpful for the community to continue conducting more research on this topic.\n\nWEAKNESSES:\n1. While I appreciate the comprehensiveness of the experiments on the proposed benchmark, I think the author fails to provide an informative analysis of the results. Currently, the results simply show that by optimizing the hyperparameters in the unified framework, we obtain better performance, which is a natural outcome since the existing methods are included in the framework. From a research perspective, I can think of several questions worth investigating, such as whether the optimal hyperparameters vary between tasks and, when using fixed models, how the optimal hyperparameters are affected by the target objective function. With this investigation, it would be best to reach some general conclusions to guide users in tuning the hyperparameters in practice, rather than just relying on grid search.\n\n2. It seems that for around half of the tasks, the improvement is only marginal, while for a few other settings, the improvement is significant. I think this phenomenon is worth further investigation to understand in what scenarios TFG will provide improvements.\n\nQUESTIONS:\n1. Why does the highest MO energy in Table 3 show a negative improvement? MPGD should be a special case of TFG. Is this because the grid-searched hyperparameters for TFG are not optimal?\n\n2. Can the author comment on other lines of work [1,2,3,4], specifically direct optimization approaches, on training-free optimization of diffusion models with a target objective? The task setting is the same as training-free guidance. Although it is evident that the direct optimization approaches are slower than training-free guidance, it is not clear how their performance compares.\n\n[1] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7280–7290, 2023b.\n\n[2] Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. arXiv preprint arXiv:2402.14017, 2024.\n\n[3] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. arXiv preprint arXiv:2312.11994, 2023.\n\n[4] Tang, Zhiwei, et al. \"Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization.\" arXiv preprint arXiv:2405.18881 (2024).\n\nLIMITATIONS:\nSee my comments on the weaknesses above. I think there are a few unclear points worth discussing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper focuses on the unification of training-free guidance methods for diffusion models. It defines each method within a unified framework and finds that restricting the hyperparameter space is consistent with existing methods. This framework can be broadly categorized into mean guidance, variance guidance and implicit dynamics. The paper demonstrates performance improvements in several experimental scenarios.

STRENGTHS:
* Experiments were conducted on different data sets and scenarios.
* Training-free guidance, which had been developed in different ways, was unified into a single framework.
* If the code is open-sourced, it will greatly benefit the diffusion community.

WEAKNESSES:
* Algorithm 1 seems to be one of the most important parts of this paper, but it lacks a detailed explanation. An explanation of Algorithm 1 along with the corresponding hyperparameters from Definition 3.1 in Section 3.1 would greatly aid understanding.

* In Algorithm 1, the iteration part in line 8 could be explicitly described using a for loop in the algorithm, rather than as a comment.

* I believe that Theorem 3.2 and its proof are not well formulated mathematically. If the authors want to formalize it as a theorem, they need to show mathematically that the same generated distribution can be produced. In my opinion, an explanation at an analytical level would be acceptable.

QUESTIONS:
Please see the Weaknesses part.

LIMITATIONS:
They provided in the last section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-21