Gradient Guidance for Diffusion Models: An Optimization Perspective
Authors: Yingqing Guo
Hui Yuan
Yukang Yang
Minshuo Chen
Mengdi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model; gradient guidance; reward optimization; convergence rate
Primary area: diffusion_based_models

Abstract: Diffusion models have demonstrated empirical successes in various applications and can be adapted to task-specific needs via guidance. This paper studies a form of gradient guidance for adapting a pre-trained diffusion model towards optimizing user-specified objectives. We establish a mathematical framework for guided diffusion to systematically study its optimization theory and algorithmic design. Our theoretical analysis spots a strong link between guided diffusion models and optimization: gradient-guided diffusion models are essentially sampling solutions to a regularized optimization problem, where the regularization is imposed by the pre-training data. As for guidance design, directly bringing in the gradient of an external objective function as guidance would  jeopardize the structure in generated samples. We investigate a modified form of gradient guidance based on a forward prediction loss, which leverages the information in pre-trained score functions and provably preserves the latent structure. We further consider an iteratively fine-tuned version of gradient-guided diffusion where guidance and score network are both updated with newly generated samples. This process mimics a first-order optimization iteration in expectation, for which we proved $\tilde{\mathcal{O}}(1/K)$ convergence rate to the global optimum when the objective function is concave. Our code is released at https://github.com/yukang123/GGDMOptim.git.

Gradient Guidance for Diffusion Models: An Optimization Perspective
Yingqing Guo
Hui Yuan
Yukang Yang
Minshuo Chen
Mengdi Wang
Princeton University
Abstract
Diffusion models have demonstrated empirical successes in various applications and can be adapted to task-specific needs via guidance. This paper studies a form of gradient guidance for adapting pre-trained diffusion model towards optimizing user-specified objectives. We establish mathematical framework for guided diffusion to systematically study its optimization theory and algorithmic design. Our theoretical analysis spots a strong link between guided diffusion models and optimization: gradient-guidee diffusion models are essentially sampling solutions regularized optimization problem, where the regularization is imposed by the pre-training data. As for guidance design, directly bringing in the gradient of an external objective function as guidance would jeopardize the structure generated samples. We investigate modified form of gradient guidance based on a forward prediction loss, which leverages the information in pre-trained score functions and provably preserves the latent structure. We further consider an iteratively fine-tuned version of gradient-guides diffusion where guidance and score network are both updated with newly generated samples. This process mimics first-order optimization iteration in expectation, for which we proved (LL/K) convergence rate to the global optimum when the objective function is concave. Our code is released https:://githut cow//ukang122/GGGDMMMM
Introduction
Diffiusion models have emerged as a significant advancement in the field generative artificial intelligence, offering state-of-the-art performance image generation [58, 56, 22]. These models operate by gradually transforming random noise into structured output, utilizing the score function learned from data. One of the key advantages of diffusion models is their flexibility which allows controlled sample generation for task-specific interest, excelling diffusion models wide range of applications, such as content creation, sequential decision making, protein engineering [38, 1, 29, 11].
Controlling the generation of large generative models stands at the forefront of AI. Guidance and fine-tuning are two most prevalent approaches for controlling the generation of diffusion models. Unlike fine-tuning which changes the weights of pre-trained models, guidance mechanism enables a more directed and flexible control. Adding gradient-baser guidance during inference pioneered by classifier guidance [60, 22], which involves training -eeeddepndeent classifier. Diffusion Posterior Sampling (DPS) [16] introduced a fully training-fire free form gradient- based guidance, which removes the dependence on time. This method has since been explored in numerous empirical studies [17, 37, 43, 65, 57, 70, 4, 35). However, despite these empirical successes, significant gaps
Equul contribution. Department of Electrical and Computer Engincering, Princeton University. Authors' emails are: (yg6736, huiyuan, yy1325, minshuochen, mengdix)@princeten edu.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

remain in the theoretical understanding and guarantees of gradient-based guidance in diffusion models.
Problem and Challenges Suppose we have a pre-trained diffusion model that can generate new samples faithfully, maintaining the latent structure of data. We study the problem of adapting this diffusion model to generate new samples that optimize task-specific objectives, while mainlaining the learned structure in new samples. This problem has a strong connection to classic optimization, guided diffusion offers new possibilities to optimize complex design variables such as images, videos, proteins, and genomes [7, 66, 42] in generative fashion. More comprehensive exposure this middle ground can be found in recent surveys [69, 12, 30].
Given the optimization nature of this problem, it's critical to answer the following theoretical questions from an optimization perspective: (i) Why doesn't It simply applying the gradient of the objective function w.r.t. the noised sample work? (ii) How add guidance signal to improve the target objective without compromising the sample quality? (iii) Can one guarantee the optimization properties of new samples generated by guided diffusion? (iv) What are the limits of adaptability in these guided models?
Gradient-Guided Dittusion for Generative Optimization
dx.
X1" Au (X(" TT G(X)) 9 de dIV.
(optional) updata
Prentaised Difusionn
Oukted Diflusion
Adapted Oiftzaton Feid.1 Go
Weighted Fine-tuning
G
Weigtt
Previsiieed Dithationn
s7p
Compute Gradkent Gurdence We, [),
Dg
Figure 1: Gradient-guided diffusion model generative optimization, with or without adaptive fine- tuning. A pre-trained diffusion model is guided with an additional gradient signal from an external objectives function towards generating near-optimal solutions.
Scope and Contribution. In this paper, investigate guided diffusion from an optimization perspective. To answer the four questions above, we propose an algorithmic framework, see Figure 1 for an illustration. Our main contributions are summarized as follows:
Study diructure-preserving guidance. We study the design of guidance under structural data distribution belonging a latent low-dimensional subspace (Assumption 1). We diagnose the failure of naive gradient guidance and study the theoretical aspects of guidance based on forward prediction loss (Definition 1), which provably preserves any low-dimensional subspace structure (Theorem l). Establish a mathematical framework of guided diffusion. We build a mathematical framework for guided diffusion, which facilitates algorithm analysis and theory establishment. We propose and analyze an iterative guided diffusion using gradient queries new samples (Algorithm 1; Figure 1 with fine-tuning block off). We give the first convergence theory showing generated samples converge to a regularized optimal solution w.r.t the objective (Theorem 4) with linear score class (12). The regularization is imposed by the pre-trained diffusion model, revealing a fundamental limit of adapting pre-trained diffusion models with guidance.
Provide rate-matching optimization theory. Furthermore, we propose an adaptive gradient- guided diffusion, where both pre-trained score network and guidance are iteratively updated using self-generated samples (Algorithm 2; Figure with the fine-tuning block turned on). We show in expectation its iteration converges to global optima within the latent subspace, at rate of O(1/K) (Theorems 3, K is # of iterations). matching the classical convergence in convex optimization. Provide experimental justification. Simulation and image experiments are provided in Section 7 to support theoretical findings on latent srructure-preserving and optimization convergence.
2 Related Works
To summarize the related work, we first give an overview of empirical studies relevant to our objective. We then discuss the theory of diffusion models, to which our main contribution is focused. Other

---

related topics, such as direct latent optimization in diffusion models and a detailed review of sampling and statistical theory of diffusion models, are deferred to Appendix A.
Classifier Guidance and Training-free Guidance. [22] introduced classifier-based guidance, steering pre-trained diffusion models towards particular class during inference. This method offers flexibility by avoiding task-specific fine-tuning, but still requires training time-dependent classifier. Training-free guidance methods [16, 57, 70, 4, 32, 50, 28] eliminate the need for time- dependent classifier, using only off-the-shelf loss guidance during inference. [16, 57, 32, 28] is a line of works solving inverse problems image and [70, 50] aims for guided/conditional image generation. Though not originally developed for solving optimization problems, [16, 70] both propose a similar guidance to ours: taking gradient on the predicted clean data Zp with respect to corrupted Ti- Differently, our paper presents the first rigorous theoretical analysis of this gradient- based guidance approach. Furthermore, propose algorithm that iteratively applies the guidance as a module to the local linearization of the optimization objective, demonstrating provable convergence guarantees.
Fine-tuning of Diffusion Models. Several methods for fine-tuning tuning diffusion models to optimize downstream reward functions include RL-based fine-tuning [7, 26] and direct backpropagation to rewards [18, 52, 68, 63]. However, these approaches often suffer from high computational costs and catastrophic forgetting in pre-trained models. Our guidance method is training-free and applied during the inference phase, eliminating the need to fine-tune diffusion models.
Theory of Diffusion Models. Current theory works primarily focus on unconditional diffusion models. Several studies demonstrate that the distributions generated by diffusion models closely approximate the true data distribution, provided the score function is accurately estimated [20, 2, 39, 13, 40, 15, 14, 6]. For conditional diffusion models, [71, 27] establish sample complexity bounds for learning generic conditional distributions. Our novel analysis establishes a connection between the sampling process in gradient-based guided diffusion and proximal gradient step, providing convergence guarantees.
3 Preliminaries: Diffusion Models and Guidance
Score-based diffusion models capture the distribution of pre-training data by learning sequence of transformations to generate new samples from noise [60]. A diffusion model comprises forward and backward process, for which we give review follows.
The forward process progressively adds noise to data, and then the sample trajectories are used to train the score function. The forward process initializes with Xo R R random variable drawn from the pre-training data D. It introduces noise via an Censten--Uhlenbee process, i.c.,
Forward Process.
1 dX, = -00))))) + Va()dw, for q(() > 0,
where (We) is Wiener process, and gt)) is non-decreasing. X, represents the noise-corrupted data distribution at time t. The conditional distribution X, zo is Gaussian, i.c,, (N((()x h(t) with (xt) exp(- fe 1 29()) (ss) and (tt) 1- a (t). In practice, the forward process will terminate at large time T so that the marginal distribution of Xx is close to (0, ID).
Backward Process. If reversing the time of the forward process, we can reconstruct the original distribution of the data from pure noise. With (W W))2 being another independent Wiener process, the backward SDE below [3] reverses the time in the forward SDE (1),
1 X 2
dx
V Vlogpr-((XF) dt + dWW
(2)
Here pr()) denotes the marginal density of X; in the forward process. In the forward SDE (2), the score function V Pe())()) plays a crucial role, but it has to be estimated from data.
Score Matching. To learn the unknown score function V log (-), we train a score network s0(2, t) using samples from forward process. Let D denote the data for training. Then the score network is
3

---

learned by minimizing the following loss:
min !,"
EngEDEass [VV,, log @<(2)/20)) s(x,, t)]l"] dt,
where a given function class, Ep denotes the empirical expectation over training data and Reg,/og (mo denotes condition expectation over the forward process, (e (20) is the Gaussian transition kernel, i.e., (2th(1)) -D/2 exp(- (-1 alI)zoll ((()))ol//(2(())))
Generation and Guided Generation. Given a pre-trained score function 8g, one generates samples by the backward process (2) with the score replaced by sg. Further, can add additional guidance to steer its output distribution towards specific properties, as formulated in Module 1.
Module Guiiddd Baaeeed_BBcckaardsaaeeee.e. G)
1: Input: Score Sa, guidance G default to be zero for unguided generation.
2: Hyper-parameter: T.
3: Initialized X3 A0, I), simulate the following SDE till time T:
dxt 2 xt + (xt G . T 11 + dW.
4: Output: Sample X
A common goal of guided generation (Module 1) is generate X with desired property Y = from the distribution P(X|Y = y). To this end, essentially needs to learn the conditional score function VA, log Pe (x+ y) The Bayes rule gives
V se log (z y) v Vlogp((Z1)+ V., log Py xx)
(4)
est. by (m).t)
by guidance
When a pre- pre-trained score network so (Z,, t) = V (e what remains is to estimate Vz, log ze) and add it as the guidance term G to the backward process in Module 1.
Classifier and Classifier- Guidance. Classifier guidance [60, 22] samples from P(X = y) when Y is discrete label. This method estimates Vz, Pegp (re) by training auxiliary classifiers, denoted as P(y It, t), and then computing the gradient of the classifier logits as the guidance, i.e., G( t) = V., log p(y 24, t), Alternatively, classifier-fres guidance [33] jointly trains a conditional and an unconditional diffusion model, combining their score estimates to generate samples.
Notations. For random variable X, P's represents its distribution, and p(x) denotes its density. For X, Y jointly distributed, P(X Y y) denotes the conditional distribution, and p(x |y) its density. The conditional expectation is denoted as E[æ y] Let DD be the pre-training data, and let Ep be the empirical expectation over D. The empirical mean and covariance matrix are ju == E [x] and Σ = EuED (æ (= ū T. For matrix A, Span(A) denotes the subspace spanned by its column vectors, and for a square matrix A, A denotes its inverse or Moore-Penrose inverse. For any differentiable function f _ + T Vf p denotes Jacobian matrix, i.c., (Vf)ig 8ag
0f:(w)
A Primer on Gradient Guidance
Let us start with stating the problem we want to study: suppose given a pre-trained diffusion model where the
network sg(x,, f) provides good approximation to the score function V log p(rz), ), the goal is to generate novel samples with desired properties that can be measured by ussrr-pecifiie differentiable function f. We will refer to f as a reward or objective function later on. To achieve this goal, from Section 3, we know that guided generation with diffusion model is good candidate, which deploys the following guided backward process (Module 1):
dx = X (X
()+G(X; , 1) di + dW..
2
Here the guidance term G what focus on and wish design. Specifically, we want construct this guidance term G based on the gradient Vf of general objective f. This is motivated by the

---

gradient methodology in optimization, a natural, intuitive way for adding guidance is to steer the generated samples towards the steepest ascent direction of f [16, 4, 18].
4.1 Structural Data Distribution with Subspace
When incorporating property optimization the generation process, it's crucial to consider intrinsic low-dimensional structures of real-world data, such as local regularities, global symmetries, and repetitive patterns [62, 55, 51J. Blindly improving f at the cost of losing these structures degrades sample quality dramatically. This quality degradation, also known as "reward over-optimization,, is common challenge for adapting diffusion models towards an external reward [71, 63].
To study the design of guidance that mitigates the risk of over-optimization, admits low-dimensional latent subspace, formulated in the following assumption.
focus on data that
Assumption (Subspace Data). Data X E RD can be represented as X = AU, where A E Rxd an unknown matrix with orthonormal columns, and the latent variable U E TRé follows some distribution P. with density Pu- Here d & D. The empirical covariance of U is assumed full rank.
In the rest of this section, we investigate the principles for designing a guidance based on the gradient of f that ensures (i) improving the value of f. and at the same time, (ii) being adhere to the subspace structure, i.e. generated samples being close to the subspace spanned by A.
4.2 Naive Gradient Does't Work as Guidance
tempting simple choice of the guidance is by taking the steepest ascent direction Vf, which we refer as naive gradient guidance i.e.,
a(X ,)) & V)(F)).
(5)
However, the naive gradient guidance (5) would jeopardize the latent structure of data, which is demonstrated by the following proposition:
Proposition (Failure of Naive Guidance). For naive guidance o(X; .t) b(t)V suppose
b(t) > bo > 0 for t > to- For data in subspace under Assumption I and reward f(z) ggd (A) with h(t) = 1 1-exxp(- exp(-v/t, then the off-subspace component of the generated sample consistently large:
E = Cg, C' > exp (-5/2)60-
The intuition provided by Proposition is, while the trained score network effectively steers the distribution toward the latent subspace [10], the gradient vector f may point outside the subspace, causing the generated output to deviate from it (Figure 2). This is why naive gradient guidance fails. [63] also observed this, explaining that V is not computed for = T' i.c. it is not aligned with the clean data space.
Hnwent
Baplo
Gudliens
Node
Protrabed
Figure 2: Directly adding the gradient of the objective function to the backward process sabotages the subspace structure. Left: Directly adding gradients that point out of the data subspace causes samples to leave the subspace. Right: Numerical experiments show that naive gradients lead to substantially larger off off-subspace error compared to our gradient guidance Gioss (Definition 1); see Section for experiment details.
4.3 Motivating Latent Subspace Preserving Guidance from Conditional Score Function
Failure of the naive gradient in maintaining data structure motivates us to seek alternatives. To get some inspiration, we start with the most elementary Gaussian probabilistic model and linear f. Later we will drop this assumption and consider general data distributions and general f.
Assumption 2 (Gaussian Linear model). Let data follow a Gaussian distribution, i.e., X NN, E), and let fz) = = x be linear function for some g E RD, Let Y f(X) + e with independent, identically distributed noise NO, o²) for some 0.
5

---

By the Bayes' rule, the conditional score Vx, Pe y takes the form of a sum given by (recall (4))
VR, log (z y) = V log (og))(())= + Vx, log Pe xz).
est. by sp(
to be est. by guidance
Under the Gaussian assumption, we derive the following closed-form formula of the guidance term Pr(y mx) that we want to estimate. The proof is provided in Appendix D.4.
Lemma (Conditional Score gives Gradient like Guidance). Under Assumption 2, we have
Va log (y m4) = (m)) Va, (u - 3 =x))²
(6)
where E[z x denotes the conditional expectation of zo given xı in the the forward process (1), and (2)) is the variance of the conditional distribution Xx It.
The form of conditional score shown in Lemma
motivates proposed gradient guidance:
Definition I (Gradient Guidance of Look-Ahead Loss). Given gradient vector 9, define the gradient guidance of look-ahead loss as
Gloss (Zt, t) == -(()) 2z, (y- g B[rol=z))
(7)
where B(1) > 0, R are tuning parameters, and [8 [Tr]] is the conditional expectation of xo given in the forward process (1), i.e., dX,
(t) X, dt + /g(t)
The formula in (7) generalizes the intuition of a conditional score for any data distribution and objective function. It scales with the residual term 9 E[ro xre). tuning the strength of guidance. Here, E[z, 2z]] represents the expected clean data To given I; in the forward process, which coincides with the expected sample in the backward view. This residual measures the look-ahead gap between the expected reward of generated samples and the target value. The look-aheal loss (yy - g [[ [xx]) resembles the proximal term commonly used in first-order proximal optimization methods.
Remark. The gradient guidance (7) aligns with the groundtruth conditional score in (6) under the assumptions of Gaussian data and linear reward (Assumption 2). This theoretical motivation, rooted fundamental framework, distinguishes our work from the empirical practice, such as DPS [16] and universal guidance [4].
key advantage of Gtoss is that it enables preserving the subspace structure, for any data distribution under Assumption 1. This is formalized in the following theorem, the full proof in Appendix D.5.
Theorem (Faithfulness of Giose to the Low-Dimensional Subspace of Data). Under Assumption 1, it holds for any data distribution and g R that
Gioso f) E Span(A).
(8)
Proof Sketch.
We have
Na (y g E(xo xx,))² V E[zo xx]
We will show that the Jacobian V 120 [x]] maps any vector 9 RD to Span(A). To see this, we utilize the score decomposition result in Appendix D.I and plug it into the equality B[ro|==]
a (1) (Ir + h(t) px,))) (Tweedie's formula [24]), we have
B[ra xx] - (t) (z+ + h(t) [Am(A xx) h (t)=.)) = h(t)/a(t) Am(A Am));
(9)
here m(u) = V log pt (u) + 1h-((t),,, Pu)( latent density (Appendix D.1). We see Va, E[zo| maps any vector Span(A) (A) because m(-) takes A A x, as input in the expression of E[zo [r).
We highlight that the faithfulness of GLoes holds for arbitrary data distribution supported the latent subspace. It takes advantage of the score function's 's decomposition (19), having the effect of automatically adapting onto the latent low-dimensional subspace data.
We provide rigorous guarantee for manifold preservation of gradient guidance, property previously discussed by [16, 17, 32]. However, while [16, 17] claim manifold preservation, they do not present formal mathematical proof. [32] relies heavily on pre-trained autoencoders for manifold projections, which are often unavailable in practical scenarios.
Remark.

---

4.4 Estimation and Implementation of Gloss
In this section, we discuss the estimation and computation of Gloss based on a pre- pre-trained score function se in practice. Gloss involves the unknown quantity E, [| One can construct estimate h
E[ [te] by considering the Tweedie's formula [24]: V (x
h (t) [ate ((t) [+
which gives rise to
E[zg| := a (t) (a, + (2,))
(10)
and we refer it as the look-ahead estimator. The estimator (10) is widely adopted in practice [56, 4]. Here att) and h(t) are the noise scheduling used in the forward process (1).
Thus, we have obtained an implementable version of the gradient guidance Gloss, given by
Gloss (zt, t) = - B(t) Vv, - (a= (at + hot)ss (xx,))))) With slight abuse of notation, we use Gloss refer this im- plementable formula (11) in the remainder of this paper. Here,
(11)
Compuse Riaalss
y is a target reward value from conditional score analysis under a Gaussian model and is treated as a tuning parameter in prac- implement.
Square Losa "[
tice. The gradient guidance (11) is lightweight
*pfoi. 4)
Given a pre-trained score function 8p in the form of a neural network, computing (11) involves calculating the squared loss
Gradient wat
(v g 30-9"Ejen E[2o|=:]) via forward pass of so and a backward pass using the -roodggadiettt feature of deep-leaning frameworks such PyTorch and TensorFlow. See Figure for illustration.
Figure 3: Computing Gloss-
5 Gradient-Guided Diffusion Model as Regularized Optimizer
In this section, we study if gradient guidance steers pre-trained diffusion models to generate near- optimal samples. Our results show that: 1) Iterative gradient guidance improves the objective values; 2) The pre-trained diffusion model acts as a regularizer from an optimization perspective.
5.1 Gradient-Guided Generation with A Pre-trained Score
Assuming access pre-trained score network so and the gradient of the objective function f, present Algorithm I to adapt the diffusion model and iteratively update the gradient guidance (11). See Figure for illustration.
Alg. takes any pre-trained score function 8o as input. Each iteration evaluates V)(()) at samples from the previous iteration (Line 5(i)), computes the gradient guidance Gloes with the new gradient (Line 5(i)), and generates new samples using the updated guidance (Module 1). The algorithm outputs an adapted diffusion model specified by (sp, Gx).
Algorithm Gradient-Guiie Diffusion for Generative Optimization
1: Input: Pre-trained score network so (-))), differentiable objective function f.
2: Tuning Parameter: Strength parameters (t),, {yu T number iterations K, batch sizes Bu
3: Initialization: G. = NULL.
4: for k = K
do
5: Generate: Sample ž Guided_ Baadeed_BackvaardSapleeee. Gx) using Module 1, for i E [Ba].
6: Compute Guidance:
(i) Compute the sample mean Zμ := (1/Ba) > Z4.6.
(ii) Query gradient 9 (z
(iii) Update gradient guidance G (.,.) Glose (..)) via (7), using so, gradient vector gie, and parameters
Y/ and ((t).
7: end for
8: Output: (se, Gx).
5.2 Gradient-Guided Diffusion Converges to Regularized Optima in Latent Space
We analyze the convergence of Alg. and show that in final iterations, generated samples center around a regularized solution of the optimization objective f within the subspace Span(A). Our theorems allow the pre-training data to have arbitrary distribution.
Assumption 3 (Concave smooth objective). The objective f IR." + R. concave and L-smooth w.r.t. the Bemii-norm Il 1. i.e., fx1) (x2))ls S L || xa ralls 1 for any Tı, Z2.
7

---

While Alg. I works with any pre- vre-trinned score network, we study its optimization properties focusing on the class of linear score functions given by
8 = {s(2, Cax + be E b, ER²).
(12)
Remark on the linear parametrization of score network (12): Analyzing the output distribution
of guided diffusion is challenging because the additional guidance term destroys the dynamics of reverse SDE. A linear score is a natural and reasonable choice for characterizing the output distribution, as it also adopted by [44].
With linear score function (12), pre-training a diffusion model is equivalent to using Gaussian model to estimate and sample from the estimated distribution. Thus, the guidance Gloss is also linear in Zy, and the final output follows a Gaussian distribution; see (25) in Appendix E. We focus on the mean, Pk. of the generated distribution from the backward sampling of (sg, Gk) (as T xc), and establish its optimization guarantee.
Theorem (Convergence to Regularized Maxima in Latent Subspace in Mean). Let Assumptions I the score function class (12) for pre-training and computing guidance. Then A gives an adapted diffusion model that generates new samples that belong to Span( A). Further, for any X > T., there exists B((),, {yk) and batch size Bk, such that with high probability 1 6, the mean of the output distribution P.K converges be near A. and it holds
and 3 hold. Suppose
K
(((xx))((pk)== f f f((x) - f(
O dlog
where A, x is an optimal solution of the regularized objective:
{
AAA argmax =ESpan(A)
(13)
where Jt, E. are empirical mean and covariance of pre-training data DD.
2 - (23) centers the data's mean jı and is stronger in directions where the original data has low variance. Thus, the pre-trained score acts as "prior" in the guided generation, favoring samples near the pre-training data, even with guidance.
Remarks. (1) The regularization term
(2) The regularization term cannot be arbitrarily small, as our theorem requires À L. Thus, only adding gradient guidance cannot achieve the global maxima. If the goal is global optima, the pre-trained score must be updated and refined with new data, as explored in Section 6.
(3) The convergence rate is lincar in the latent dimension d, rather than data dimension D. Since Gloss is faithful to the latent subspace (Theorem 2), the generated samples and optimization iterates of Alg. remain within Span(A). This leverage of the latent structure results in faster convergence.
Gradient-Guided Diffusion with Adaptive Fine- Sine-Tunigg for Global Optimization
In the previous section, we have seen that adding guidance to a pre-trained diffusion model can't improve the objective function unlimitedly due the pre-trained score function acting regularizer. We consider adaptively fine-tuning pre-trained diffusion models to attain global optima. Empirically, fine-tuning diffusion models using self-generated samples has been explored by [7, 18).
6.1 Adaptive Fine-Tuning Algorithm with Gradient Guidance
We propose an adaptive version of the gradient-guided diffusion, where both the guidance and the score are iteratively updated using self-generan reffreeeeette samples. The full algorithm is given in Algorithm 2.
We introduce weighting scheme fine-tune the score network using mixture of pre-training data and newly generated samples. In Round k, let Di...., Dj be sample batches generated from the previous rounds. Let )/ be set of weights. Conceptually, at Round ks we update the model by minimizing the weighted score matching loss:
/' Σ 1=0
min #ES
Wk. E.o. ED. Exiizo
[Vz, log (x.laa))) - s(x,, 1)][ dr,
(14)
where Do := the pre-training data, For illustration, please see also Figure 1, and the practical implementation of Alg. 2 is in Appendix F.
8

---

Algorithm 2 Gradient-Guided Diffusion with Adaptive Fine-tuning
1: Input: Pre-trained score se(;-), differentiable objective function f.
2: Tuning Parameter: strength parameter ((t), (yk [== weights (u K-1 number of iterations
K. batch sizes [Bx}-
3: Initialize: se, Sp. Go NULL.
4: for k 0,... K
5: Generate: Sample batch Dr {2t ) from Guided_ Bacieeed_BecccaardSSiieeeees Gk) (Module 1).
6: Compute Guidance:
(i) Compute sample mcan ãs (1/Be) E Sk.i, and query gradient ge (āe)..
(ii) Update s0, to 80p-1 by minimizing the re-weighted objective (14).
(iii) Compute Gx+1(..-) (:x)) ) in (7), using 88411 and ge. with parameter yke ((t).
7: end for
8: Output: (s0x Gx).
6.2 Guided Generation Finds Unregularized Global Optima
Finally, we analyze the optimization properties for gradient-guidet diffusion model with iterative finetuning. We establish that the process of Algorithm 2 yields final output distribution whose mean, denoted by μuk. converges to the global optimum of f.
For simplicity of analysis, we study the following function class
s' s'= ((x, Caa + b, be E RD).
(15)
where Cg is set to stay the same in pre-trained scores, with only b; updated during iterative fine-tuning.
Theorem 3 (Convergence to Unregularized Maxima in Latent Subspace in Mean). Let Assumptions I and 3 hold, and assume there exists M > such that <'A,> < M for all X 2 0. Suppose we use the score function class (12)for pre-training 89 and the class (15) for finetuning it. Then Algorithm gives adapted diffusion model that generates new samples belonging to Span(A). Further, there exists {3(t)}, [ynt, { and i. such that with probability 1 8,
[LL [dI. log K
K
fa - f(uk)= =0
log
(16)
K
where l = max (] E (A))
Theorem 3 illustrates that fine-tuning a diffusion model with self-generated data can reach global optima while preserving the latent subspace structure. The convergence rate matches standard convex optimization in terms of gradient evaluations, K. Compared to standard gradient solvers, guided diffusion models leverage pre-training data solve optimization problems in low-dimensional space, preserving desired structures and enabling more efficient exploration and faster convergence.
7
Experiments
7.1 Simulation
We conduct numerical simulations of Algorithms 1 and 2, following the subspace setup described in Assumption 1. Specifically, set d 16, D 64, The latent variable is drawn from A Ia) and used to construct
Au, where A is randomly generated orthonormal matrix. We define the objective function [(x) 10- (0 TT 3)2. To approximate score function, we employ a version of the U-Net [54] with 14.8M trainable parameters. More details including how set up
are provided in Appendix F.1.
Preserving Subspace Structure. We first demon- strate that Glose preserves the subspace struc- ture learned from the pre-trained model. For comparison, also tested the naive guidance G(z,,t) t) := B(t) (y g "[]ool=))) g (more details in Appendix F.I.). Figure 4 (left) shows that Gloss performs much better than the naive gradient G in preserving the linear subspace. Figure 4 (right)
Perrared
Peetraned
Figure Comparison between two types of gradient guidance G and Gjoss (left: Alg. 1; right: Alg. 2). The off/on support ratio of the generated samples is defined as [xx
|

---

demonstrates that off- errors increase with adaptive score fine-tuning (Alg. 2) due to distribu- tion shift, with G resulting in more severe errors than Gloss.
Convergence Results. Figure 5 (a) and (b) show that Alg. converges to a sub-optima, objective value, leaving gap to the maximal value. This aligns with our theory that the pre-trained model acts regularizer in addition to the objective function. Figure 5 (c) shows that Alg 2 converges the maximal value of the objective function. As illustrated by Figure (d), samples from Alg. 1 mostly stay close to the pre-training data distribution (dotted contour area), whereas samples from Alg. move outside the contour as the diffusion model fine-tuned with self- seff-generated data.
rounde
Iound
round
(a) Alg. 1:0= A3
(b) Alg. 1:
10 110
9 (c) Convergence of Alg. 2 (d) Visualizing sample gener-
attion
Figure 5: Convergence of Algorithms and 2. (a) and (b) under different for the objective function. (d)
visualizes the distribution of the generated samples Alg, (red) and Alg. (blue) across the iterations.
7.2 Image Generation
We validate our theory in the image domain for Algorithm We employ the StableDiffusion v1.5 model [53] as the pre-trained model. For the reward model, we follow the approach outlined by [71] to construct synthetic model. This model is based on a ResNet-18 [31] architecture pre-trained on ImageNet [21], with the final prediction layer replaced by randomly initialized linear layer that produces scalar outputs. For more experiment details, refer Appendix F.2.
'rabbit
Seweard
horre
TOx
10
Iteration
Borabion
Figure 6: Reward increase and effect images across iterations. Left: Reward increases and converges across iterations. Larger guidance strength (smaller regularizer strength) results in higher convergent reward value. Right: Images become more abstract, shifting from realistic to virtual backgrounds as reward increases.
Results. By Algorithm 1, the reward increases and converges. Figure (left) shows the reward changes with optimization iterations. The hyperparameter tunes the strength of guidance and is inversely related to the strength of the regularizer (theoretical implications in Appendix E). A larger guidance strength (smaller regularizer strength) leads to a higher convergent reward value. Figure (right) illustrates the changes in generated images across iterations. As the reward increases, the images become increasingly abstract, shifting from photo-realistic with detailed backgrounds to more virtual, stylized ones.
Conclusion
In this paper, we focus on gradient guidance for adapting or fine-tuning pre-trained diffusion models from optimization perspective. We investigate the look-ahead loss based gradient guidance and two variants of diffusion-based generative optimization algorithms utilizing it. We provide guarantees for adapting/fine-tung weafffnnnetttnnggneeeeee diffusion models to maximize any target concave differentiable reward function. Our analysis extends linear subspace data, where our gradient guidance and adaptive algorithms preserve and leverage the latent subspace, achieving faster convergence to near-optimal solutions.
10

---

Acknowledgments
Mengdi Wang acknowledges the support by NSF IIS- IIS-2107344, NSF CPS-2312093, ONR 1006977 and Genmab.
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision mecision-making? arXiv preprint arXiv:2211. 15657, 2022.
[2] Michael S Albergo. Nicholas M Boffi, and Eric Vanden- Sunden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprini arXi::2303.08797, 2023.
[3] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 1233):313-326, 1982.
[4] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
[5] Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. arXiv preprint arXiv:2402.14017, 2024.
[6] Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis, Linear convergence bounds for diffuuson models via stochastic localization. arXiv preprint arXiv:2308.03668 2023.
[7] Kevin Black, Michael Janner, Yilun Du Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
[8] Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107 2020.
[9] Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and TrendsO in Machine Learning, 803-4:231-337,,200
[10] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.007194 2023.
[11] Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. Opportunities and challenges of diffusion models for generative ai. National Science Review, page nwae348, 2024.
[12] Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. An overview of diffusion models: Applications, guided generation, statistical rates and optimization. arXiv preprint arXiv:2404.07771, 2024.
[13] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209. 12215 2022.
[14] Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. arXiv preprint arXiv:2305.1 /798, 2023.
[15] Sitan Chen, Giannis Daras, and Alex Dimakis. Restoration-degradatioo beyond linear diffusions: A non-asymptotic analysis for ddim-type samplers. In International Conference Machine Learning, pages 4462 4484. PMLR, 2023.
[16] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems, arXiv preprint arXiv:2209.1460 2022.

---

[17] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683-25696 2022.
[18] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
[19] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.055314 2022.
[20] Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:17695-17709, 2021.
[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-2288 228.255.22 Icee, 2009.
[22] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[23] Zehao Dou, Minshuo Chen, Mengdi Wang, and Zhuoran Yang. Theory consistency diffusion models: Distribution estimation meets fast sampling. In Forty-first International Conference on Machine Learning.
[24] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496);1602-1614 2011. 2011.
[25] Ahmed El Alaoui, Andrea Montanari, and Mark Sellke. Sampling from mean-field gibbs measures via diffusion processes. arXiv preprint arXiv:2310.08912 2023.
[26] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305. 2023.
[27] Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arXiv preprint ar 1981 2024.
[28] Tomer Garber and Tom Tirer. Image restoration by denoising diffusion models with iteratively preconditioned guidance. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 25245-25254, 2024.
[29] Nate Gruver, Samuel Stanton, Nathan C Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. arXiv preprint 2Xiv:2335520000 2023.
[30] Zhiye Guo, Jian Liu, Yanli Wang, Mengrui Chen, Duolin Wang, Dong Xu, and Jianlin Cheng. Diffusion models in bioinformatics: A new wave of deep learning revolution in action. arXiv preprint 230.:::2002.0007.. 2023.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[32] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, al. Manifold preserv- ing guided diffusion. arXiv preprint arXiv:231 166424 2023.
[33] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 127558200. 2022.
12

---

[34] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[35] Kaixuan Huang, Yukang Yang, Kaidi Fu, Yanyi Chu, Le Cong, and Mengdi Wang. Latent diffusion models for controllable ma sequence generation. arXiv preprint arXiv:2409.09825 2024.
[36] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwa- janakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1534-1345 2024.
[37] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 55:23593-23606 2022.
[38] Zhifeng Kong. Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: versatile diffusion model for audio synthesis. arXiv preprint drXiv:2009.09761, 2020.
[39] Holden Lee, Jianfeng Lu, and Yixin Tan, Convergence for score-based generative modeling with polynomial complexity. arXiv preprint 2261::22006.62227,, 2022.
[40] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score- score-based generative modeling for general data distributions. arXiv preprint S2KXV::2209123881. 2022.
[41] Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208. /4699, 2022.
[42] Yixin Liu, Kai Zhang. Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology. limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024.
[43] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[44] Pierre Marion, Anna Korba, Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Amaud Doucet, Felipe Llinares-López, Courtney Paquette, and Quentin Berthet. Implicit diffusion: Efficient optimization through stochastic sampling. arXiv preprint arXi:22402.05468, 2024.
[45] Song Mei and Yuchen Wu. Deep networks denoising algorithms: Sample-effcciet learning of diffusion models in high-dimensional graphical models. arXiv preprint arXiv:2309.11422, 2023.
[46] Andrea Montanari and Yuchen Wu. Posterior sampling from the spiked models via diffusion processes. arXiv preprint arXiv:2304.1 //49, 2023.
[47] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. Proceedings of the International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[48] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. arXiv preprint arXiv:2303.01886, 2023.
[49] Jiachun Pan, Jun Hao Liew, Vincent YF Tan, Jiashi Feng, and Hanshu Yan. Adjointdpm: Adjoint sensitivity method for gradient backpropagation of diffusion probabilistic models. arXiv preprint ar0iv:2307. 10711, 2023.
[50] Jiachun Pan, Hanshu Yan, Jun Hao Liew, Jiashi Feng, and Vincent YF Tan. Towards accurate guided diffusion sampling through symplectic adjoint method. arXiv preprint 1rX2v:2312.12030. 2023.
13

---

[51] Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.0880 2021.
[52] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXlv:2310.037.9, 2023.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[54] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[55] Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2323-23333 2000.
[56] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[57] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference Machine Learning, pages 32483-32498, PMLR, 2023.
[58] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[59] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
[60] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13356, 2020.
[61] Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung- Hui Chang. Tuning- alignment of diffusion models with direct noise optimization. arXiv preprint arXiv:2405.1888 18881, 2024.
[62] Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, (90(5500):231923333 2000.
[63] Masatoshi Uchara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine- tuning of continuous-time diffusion models as entropy-regularized control. arXiv preprint arXiv:2402. 15194, 2024.
[64] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusson latent optimization improves classifier guidance. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 7280-7290, 2023.
[65] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint 22Xxv:2212.00499 2022.
[66] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian Trippe, Jason Yim, Helen Eisenach, Woody Ahemn, Andrew Borst, Robert J Ragotte, Lukas Milles, al. De novo design of protein structure and function with rfdiffusion. Nature, 62097976): 2023,
14

---

[67] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. arXiv preprint 0rXi::2402.07747,,222 2024.
[68] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint NXXv:23004009977, 2023.
[69] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[70] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23174-23184, 2023.
[71] Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. arXiv preprint arXiv:3307.07055, 2023.
15

---

A Additional Related Works
Direct Latent Optimization in Diffusion Models. Besides guidance methods, an alternative training-free route by optimizing the initial value of reverse process [64, 5, 36, 61, 49]. These methods typically backpropagate the gradient of reward directly the initial latent vector through an ODE solver, utilizing the chain rule, Thus at inference time, the reverse process is unchanged except for being fed with an optimized initialization, different from the guidance method we studied.
Sampling and Statistical Theory of Diffusion Model In contrast to the fruitful empirical advances, the theory of diffusion models still limited. To the best of our knowledge, a theoretical understanding of fine-tuning diffusion models is absent. Existing results mainly focus the sampling ability and statistical properties of unconditional diffusion models. In particular, for sampling ability, line of works shows that the distribution generated by a diffusion model is close to the data distribution, as long as the score function accurately estimated [20. 2, 8, 39, 13, 40]. The accuracy of the estimated score function is measured in terms of an Lo or L2 L2-norm distance. More recently, [15, 14, 6] develop refined and tighter analyses using Taylor expansions of the discretized backward process and localization method. It is worth mentioning that the analysis [15, 14, 6] extends to broader sample generation processes such deterministic ones based probabilistic ODEs. Going beyond distributions in Euclidean spaces, [19] analyzes diffusion models for sampling distribution supported on a -ow-dimensional manifold. Moreover, [46] consider sampling from symmetric spiked models, and [25] study sampling from Gibbs distributions using diffusion processes.
Turning towards the statistical theory of diffusion models, [59] and [41] provide asymptotic analyses, assuming a parametric form of the score function. Unfortunately, asymptotic analysis does not lead to concrete sample complexities. Later, concurrent works, [48] and [10], establish sample complexity bounds of diffusion models for estimating nonparametric data distributions. In high dimensions, their results highlight curse of dimensionality issue without further assumptions, which also appears in [67] considering kernel methods. More interestingly. these works demonstrate that diffusion models can cireumvent the curse of dimensionality issue if the data has low-dimensional structures. In the same spirit, [45] investigate learning high-dimensional graphical models using diffusion models, without the curse of dimensionality. [23] establishes statistical theory for consistency models. For conditional diffusion models. [71, 27] establish sample complexity bounds for learning generic conditional distributions. We refer readers to [12] for an overview of contemporary theoretical
progress.
B Discussion
Limitations We do not see significant limitations in our analysis. The linear score class initiates the study of gradient-like guidance. We have already demonstrated that our proposed gradient guidance preserves low-dimensional structures for any score and data distribution (Theorem 1). Moving forward, we aim to analyze the convergence properties for general score functions.
Broader Impacts This paper's 's exploration of muudient-guided diffusion models offers theoretical implications that could inspire further algorithmic advancements. By establishing a connection between gradient-guided diffusion and optimization, the research opens new avenues for efficient and effective optimization through a generative approach. This work has the potential to enhance the precision and personalization of AI systems across various fields, including image synthesis and drug design. However, these techniques could also be misused to generate harmful content.
C Characterization for Output Distribution of Backward Process
In this section, we provide analytical characterizations for the output distribution of the backward process guided by Gloss when the pre-trained score is linear. We first give the result of score matching as follows.
Lemma 2 (Pre-trame tre-training with Linear Score Functions). the class in (14) is
Suppose for pre-training the score network,
S = (s(x Ca bb: C. E R DxD b, E R
(recall (12))
16

---

If we freeze Ct in (12), that is, minimizing the score matching objective (14) over the class (s(z,t) t) = Cx + by : by E RD} gives
80( 1) C2 -
where x -(E W.ED, [z) live (14) over the class (12) yields
(Ei= Wk.s) Moreover, minimizing the score matching objec-
(a 1) ((()T) (Ix a(t)))
where Σ -Ei EEDD [( 2)(x -=)x=)))]] =) / (Ei wkss) are weighted data covariance.
Proof.
Using the linear score network class S with freezing Ct, the score matching loss (14)
into
Wk, U Ae.. [I [VV,, log (2(e/o)))))))))))))))))))))))))) s(1e, t)1 dt
1 [l-mt)
[mce) h(tt) (ot a(t)xo) C,or - by 1/2
Wk, ED, Bay,
[[(
11) (ax - a(t)xo) + (alt)C,zo + b,)
1
Wk ED, Eny Mlat))))
dt
h(t)
[1b + a(11)rolll
/ trace h(1)) (c.(ctt
(n) ) (
1 T(())
it)(")) dl,
Wk W EDDDD
0
where w
E == Wkii, equality (i) follows from computing the expectation over Gaussian distribution of ze|20- We note that b, should minimize
the conditional
E Wk, EED
|1be + att)Crzoll
for any t, which leads to
b, = =
Now, we solve Ct for the second result. Substituting be into the optimization objective (14) yields:
+ w S trace h(tt) C2
1 110) ID
1 10) dt
wk EZDED,
[la(t)C,=-a(t)))))))
C
h(t)
h(t))
[/" (t) Σ Wk
(|C + w (" trace h(tt)
CT (CT
C h(t)
C. +
1
C. + n²(t)
Io) 10)) dt.
h(t)
i=0
Taking the gradient for C₁ we get
2a (1)( wk
BRDDD, [(50 =)(0 - =) 2whh())C 2wID.
t=0
Setting the gradient above to 0, we get the solution for C as
Wi, W.DDD [[00 - x)(2n =] + M(1)))) )
C - 6 Q²(())
Σ
Therefore, the proof is completed.
When Wk.0 1, Wk. 0, E [k), Lemma reduces to the pre-traning score matching.
Corollary 1. Let D be the pre-training data. Minimizing the score matching objective (3) over the function class (12) gives
so (z,, t) = - (a²(t) (x+ (t))))
(17)
The following lemma characterizes the output distribution of the backward process guided by Gtoes when the pre-trained score is linear.
17

---

Lemma 3. If the pre-trained score sa( t) is (17), substituting the score function with sø t) (oos t) in the backward SDE (2) yields, when T + 000
Egg T
X N'W
y
Eg. E
8² + Eg
0² + gg
with ((t) assigned as B(t) g (ID a (t)\//()) -1 9) -1 training data reside in Span(A) following Assumption holds X; Span(A).
Moreover if pre-
where ~ N (0, Let Xu be the initialization of the forward process. Similar to the proof in Appendix D.4, we get
Proof. Consider Xu W (i, Σ). YY=
Xo + -
X, Y
- N a "
(1) I(t a(t)g
alt)_g σ² #+g* Egg
Thus, we get (2t, is exactly the score of marginal distribution of Xt, i... V Vlog so t). According to the proof in Appendix D.4, we get se (0 1) + Gloss (Tr, = V (x y). Thus, the backward SDE turns out to be
dx;- xt VlogpT- (v) di dw.. X W
(18)
The initial distribution Po( v) of the forward process can also be obtained by (18) where replace the initial distribution as pr (ZT y). According to the data processing inequality, we get the bound of the total variation distance between the terminal distribution pf (18) and (o y):
TV (Po.PF) TV (pr; yp)
where Po: pr short hands for Po y) and pr (FT y), and sp()) is the density for the standard normal distribution N' (0, (D). Since in the forward process,
+ when T + co, we have TV (Po. pT) + when T - Do. We complete the first part of the lemma. As for the second part, if data reside in Span(A) following Assumption
have Ā Aü and Σ AE,AT where E EE E. = BuED,M=ATA [(u ú) (u ū) J.' Thus, the covariance matrix of X is
Egg T
A E En 9g AΣn
AT
x2 g2+g Eg
02 + Σg
and due to X follows Gaussian distribution, we get X² E (A). Thus, the proof is completed.
D Additional Materials for Section 4
Contents
D.1
Score decomposition for subspace data
18
D.2
Proof of Proposition
19
D.3
Discussion for gradient- like guidance
19
D.4
Proof of Lemma
19
D.5
Proof of Theorem
20
D.1
Score decomposition for subspace data
Under Assumption the score function V Pe(x) decomposes to orthogonal parts: support component belonging to the subspace; and an orthogonal component. We recall this key result Proposition which later plays a key role in deriving subspace preserving guidance.
Proposition (Score Decomposition for Subspace Data ([10) 1, Thm. 3)). Under Assumption 1, the score function V log (x:) decomposes
V log pe (wr) AV ( - - -1) (In AAT)a
(19)
B1 SILA 2.1); suppor score
(x.1); ortho, score
where (P = de( (1lllu))) (u) du with @e(-lu)) being the density of N(a(t)u, h(t)La) for the same at) and ft(t) in the forward process (1).
18

---

D.2 Proof of Proposition 1
We give the proof of Proposition 1, which shows the failure of naive gradient guidance.
Proof. Under Assumption 1, the score can be decomposed terms parallel and orthogonal to Span(A) (Proposition 2). Applying naive guidance, we examine the orthogonal reverse process:
1 2 h(T 1)
x dt + b(t)gdt + (In - AA") dw.
Solving this SDE, we get the expectation of the final state following B[ = 5o exp (-56 h (s)dss))))))))))))))))))){)))( eTT2 t)gdt. For the schedule h(t) 1 ssp(-vt), have the coefficient of direction is larger than Le expl stpp-t/2-0 2VT) t)dt > fo exp( (xp(-5/2)hb b/2)) > 0 where we can assume T > 1. Thus, E[X;_]] * O. This means the generated sample is leaving the subspace, i.e., naive gradient guidance will violate the latent structure.
D.3 Discussion for gradient-like guidance
To further clarify the derivation of our gradient guidance, we present a variant of Lemma 1.
Proposition 3. Under Assumption 2, we have
Vx, log pr(y) = ( [[ (())))+(())))+++ Eg,
(20)
where E[ x denotes the conditional expectation of given a; in the forward process (l), and B(t) = (t)) + - (ID + a² (t)/h((t -)-11) g).
Remarks. Observe that, when Σ = I, (20) suggests the following form of guidance that is aligned with the naive gradient, i.e., the steepest ascent direction:
G(I,,t) [y g B[ro| g-
However, even for Gaussian distributions, long Σ. * I, the term of (6) is longer proportional to but becomes a pre-conditioned version of the gradient. We show the guidance above can maintain the subspace structure of data in the experiments Appendix F.
Another observation is that this guidance scales with a residual
term y 9 B[ 1z)]- In particular, the residual term y g B (200 _..] tunes the strength of guidance. Recall E[ro [ denotes the posterior expectation of clean data xo given 2y in the forward process. Thus, backward view, E[xr mtg] coincides with the expected sample to be generated conditioned on 2t. In this sense, the quantity -g g E[zo | xy measures a look-ahead gap between the expected reward generated samples and the target value. A larger absolute value of the residual means stronger guidance in the backward generation process.
A
htt
Figure 7: Plot of (()), <ett,,((t)) for
L L [0, 10) when E == I.
Under (t) = in the forward process (1), we plot the theoreti- cal choice of ((t) and a(t), to in Figure 7. In practice, the choice of a(t), h(t) can vary and they are determined by the forward process used for pre-training; and ((() can be treated as a tuning parameter to adjust the strength of guidance.
The proof of Proposition 3 included the proof of Lemma 1.
D.4 Proof of Lemma
Proof. Recall (Xt)e20 is the stochastic process from the forward process. X, d (H, g T TXo + c where e <~ (N, o²) independent with Xo- Since Xo, X, and Y are joint Gaussian, we have Y X, also follows Gaussian distribution, denoted as N (my(21), o (2))) Then, the closed form of V., log Pt (y Iι) can be derived as
y NV-my(1r) my(xt))
)
27
VR, (y xz) = Va,
Ve, log (z()).
2
oy (x1)
19

---

Due to the linearity of Y with regard to Xo. my( can be computed as
my = Elg T aid = 9 Elaro xr)- (,)()))) = Ely [x] Elg" -0 + To get the variance o (2+), compute the joint distribution (Xe, Y). In the forward process, given Xy 20, X, can be written as a(t) + Z. for Ze NO (t) independent of zo. Due the linear function assumption, we have
(21)
X; Y = T 0 D 0
To
Ze
Observing that the joint distribution of (zu, C, 24) is Gaussian, we deduce
Xx Y
[a I()
alt)_g σ² + Eg 1)
N
a(t)g"I
Thus, we get oz derivation the mean my( (21), we get
o² + g) "igg- (t)g) (t) h(t)LD) ) Eg. Together with the
Vx, logpe x2) 20
1
VxL (y g Eizo m=1)
(y g Elzo æi1) V.,. Elzo zelg.
(3)))))
To get E[zo re), we derive the joint distribution of (Xo, Xi):
Xo Xx (lotia)
- N
1
Σ (a()) ((t)E+h(t)l)
a())
Thus, we get [[(ooaa)] = + a(l) have
Σ E(x a(l)p). As consequence, we
VR, logprly mx) (3)))))
Eg. o = (²2 + ogg Eg - (t)g h(t)))) 2gg -1
(y g Elzo x1)) at)) (a²
Together with the following equality by Woodbury identity, get the result.
(t Σ h(t)
2² + g Σー11
TD
g
D.5 Proof of Theorem
Proof. We expand the derivative in Gtoss as
Glosa (xe, t) = 23( -g E[ro ær)) (V,, B[ro|==)) g.
holds [[ = a(i) (x, + h(t) (x))) Via the score decomposition under linear subspace data in Chen et al. [10, Lemma 1], we have
1
1
v (og AV loget (A 22)
(In - AAT) xt.
(D-AA))
h(t)
where D denotes the diffused latent distribution, i.e., (u')) f (re (uulluo)p (wo) dug- Recall that di is the Gaussian transition kernel of the forward process and pv. is the density of latent variable Up in Assumption
To ease the derivation, we denote m(u) V loo Vloggt
1 u. It then holds that m(A)
1
1
E B[ro|21]
at)) + h(t))
Am(A xx) h(t))
21
htt) Am(AT xz).
a(l)
20

---

As consequence, we can verify that
h(t) Vm((A A (Axx)) AT,
Ve, Bazo a(t)
where Vm xx) E Rddd is the Jacobian matrix of m at A 21. Plugging the last display into Glose we conclude that
Gloss (1t, 1) 2,3()(u [(rol))) (Va, B[rolaa)) g
le
for 4 = (3( - g _ B| (x]) and g' = (Vo,l(zo) = A0d (mm(Azz))) A
a())
Span (A). The proof is complete.
E
Additional Materials for in Sections 5 and 6
Contents
E,1 Convergence Regularized Maxima Mean
E.2 Proof of Theorem
E.3 Proof of Theorem
24
E,4
Auxiliary Lemma
25
E.1
Convergence to Regularized Maxima in Mean
Here we present a more fundamental version of Theorem 2.
Theorem 4 (Convergence to Regularized Maxima in Mean). Let Assumption 3 hold, and let the pre-training data D have arbitrary distribution with covariance matrix > 0. Suppose the score function 8g is pre-trained via minimizing the score matching loss (3) over the linear function class (12). Let Alg. I take so(-;") and f as the input. For any X >> L, there exists (B(t)), {yk), {Bx} such that, with probability 2 5, the mean of the output distribution μKK converges to near x and
f ((x))=f(x))=2) =x C O (D
K
K ))
(22)
where D is the ambient dimension of cltta, and xi is regularized maximizer of given by
{F(x) - 12-
argmax HERD
(23)
where üt E are empirical mean and covariance pre-training data D.
Proof. The proof is a special case of Theorem 2 in Appendix E.2, via setting the representation matrix A Ip.
E.2
Proof of Theorem 2
We first provide proof sketch.
Proof Sketch Solving the score matching problem (3) with linear function class (12) yields pre-trained score as follows
so tz), - (a²(t)E h(t)LD) (z, (t))p).
With proper choices of (()), gradient guidance Gloss leads to the following output distribution the end of round k:
24 --- Eg.. E o² 9 Egs
Egngr.
N p
o² g). Ege Thus, we obtain the mean of the above distribution. i.e., Pk+1 = + EVf(z)), where 2; is the empirical mean of previous samples, 77. is a stepsize determined by yu. By a rearrangement, we obtain a recursive formula
#4+1 E사 + mp
(((t)) Th -'S- (ã+ p))
(24)
21

---

We observe that (24) resembles gradient ascent update from μ4 2n to M4+1 corresponding to a regularzed optimization problem (23). In this regularized objective, the original objective f(x)
incorporates an additional proximal term with À := 1/T)- Therefore we can analyze the convergence of pi by following the classical argument for gradient optimization.
Proof of Theorem 2. Define filtration {H+ K-1 with Hy be the information accumulated after Is rounds of Alg.
Ho == o(i),
H4 == (Hx-1; (=k-1,1:...,,
-I-I.B 1.,.:::...,,,,,....... E [K].
Define the expectation of samples generated k-th round as
== E|22, M4-11, k E 1K--1 -1].
Applying Corollary 1, we get the pre-trained score as (a² + h(t)ID) -1 (x1 a(t)
(2 t)
If we set yı as follows
Vv nj (o++ + gf Egk) + + R. choose
where 7 = 1/A.
And
we
B(t) at k-round as (t)
=
( (LD a (1)E/h(t)) -1 gh-1)
Then, Lemma provides the generated
distribution in k-th round:
Egh-19h-12
N i M+ 0² 9 Egk-1
(25)
Define the empirical covariance matrix of the latent variable U as Σx B [[(u u) (u-w) u) where ù Ep(u]. Then in the subspace setting, the empir- ical mean and covariance of data X can be written as J AAT and E A respectively. The mean of the sample 2 follows
A Uc Ei HH= = AAT # n AE,AT 9h-1;
where 9k-1 = (Z and E = (1/B) 2k-1,5- We rearrange the update rule to show a gradient ascent formula as follows
1*k AAT 1 AAT (NA-1 71) +1 AE.,, A V + AS,, AT (gk-1 v(((k-
= -AA ux-1 - 4
A AΣ A (μk-1 - ( - A f(u + AΣ,A (gk-1 - 1--1---f(uu-) ((((x-1))
= =AA 4 A [ - AAA;AA - (4k-1 - x + AΣ (gk-1 vf(uu-1)).
where x 1/n. Define h(x) == f() x/2 / we have
[ = AAT Mk-1 + EVh(k=11) + Σ (yk-1 (()=))))
Recall the notation for the optimum: A under the semi-norm ||-1ls
A.A arggax,, Au M(x). We consider the distance of 14 to
| - x - =
| A'A + 1) 1E + 7/ - (f(
< 1 -'AAA nEVh(u ولار + I1 (9k-1 V (1)x=1))lls--- (26)
:11
We bound the second term 1z first. According to f is L-smooth with respect to ||-1|s- we have
I = |1 - Vf( - n2 || P P
Lemma 3 shows the distribution of 24-1,5- Therefore. according to concentration inequality for Gaussian distribution, with the probability at least 1 - 5/K,, it holds
(2K)
trace (V(2k-15)) 5-1)
2 1 < 2 2log
Bx-1
We have trace (V(26-1,)) 5-1) S trace (Σ. 5-1) d. Therefore, 12 is bounded by
1 < Mo/ /llVBB... BR-1.
22

---

where Mn : : / log (2k) d. Next, we consider the first term in (26). Since x A, is the optimum of h within (A), the gradient V,, ) is in the orthogonal subspace, i.e, (,, = 0. The first term in (26) can be written as
thus EVh(E))) 0.
13 |1 2000-2xp0 x + 2 (Vh(ux-1) - Vh(= -1|Aa-1- 1 22,,22 + +2 [h(mx-1) Vh(2),)))
+ 2 1 -
17 (Vh(Hx-1) - vh(z),a)))
Recall is f adding a ||-1|s-1 regularized term. We get h is (L + A)-smotth with respect to semi norm
I-ls.-- which is derived from f L-smooth. Also, h is X-strongly concave with respect to semi norm I--lg-1 since f is concave. According to Lemma 4, we derive
(1h-1 - AA Vh(pp-1)- - VI(EAN)) S A(L + A)
1 22
||4-1-- - All
[/(((x-1) - VI(2A,)))s
Plugin the formula of 1,, we get
if 2nd(L + 2) L 2A
||u--1 - 23A -2
+ (1²² - 2Λ
2m
||Th(ua-1) - V
L
21
Since 17 1/2, it holds n²²
>
0. Due to h (L + -)smooothnsss, we get
L L+
21)(L+11)
23Aم1 1- + (1p 1 24
1 $ L 2A
|| - x
-
(L+ - ||2|1 1 22
L L + 21
(1 - 17 (L + 2))²
|| - -x 12
thus, we get the bound of T
1 I 11
where c:= 1 - 17 (T + A)1- Combing the upper bound of Jn and 12, we get with probability at least 1 8/K, for 1 < k s K,
|Ime 10--AAAM -'A.lls- scllue-1- --'ill=-- + Mo
VBa-1
As for k 1, by similar derivation, we can obtain |11 - 2 22a1| r-1 s - induction, we get with probability at least - ((K - 1)/K)6.
- x -2
By
D-1
K-1 sk-nK-1
|| - -x 2'/.1 11,- S 1|20 - + Mo
VBr
k=1
Choose B& 2 5 4k(1 -
(11-)-- - for all k E [K- - 1], then we can get
||ux - - 5 (I - - -120->2all3s
+ M vd)
(27)
22 5
where M1 n= 2log
Since h is (T. + A)-smooth with respect to |1.115 it holds
(h(( 2'Ai' 5 L+1 A
[h(2k) - ((
|| - x x -
2
Considering that Vh(**,)) 1 Span(A) yields (Vh(=A,)) A. - =2a) ing by rearranging the equation above
O. we obtain the follow-
2 (1 - 14 12133- - -1IK - Al133-) + L+1
A
f (ax,)) f(ux) <
|| - 13 (28)
2
<
A| x'A م 15-- ||nx -ل + (L + A)
Aal||1p |||u -2aA.11--11
Substitute (27) into above upper bound, with 2n = f we have
(*)x)) - [UK) x (I. + 2) ||0 - 1| 2- + Mfd
23

---

12 11- 1-- = A|AT (ū (1==)>))
Since
is the distance within Span(A), i.e. O(d). Recall
2K
7 -= -=1/A6=11-7(45 A\I = L/A, L, and M1 = v2y 2
Therefore, we get the
final result:
K
K
- - S
I
dlog
8 ))
w.p.1 - 8.
E.3 Proof of Theorem 3
We first provide proof sketch.
Proof Sketch The proof idea similar to the proof of Theorem 2. For simplicity, we analyze the case where only the most recent sample batch Dr is merged with Do for finetuning the score function. More specifically, we let Wk, = 0 for 0 < < k and wko 1 - Wr,k- Similar to the proof of Theorem 4, we obtain a recursive update rule given by
1i++1 Σ mh Σ
Vf(=) - (1 wx.K) f/ -1 - p)],
(29)
where Σい R j& is the empirical mean of previous samples. This update rule also closely resembles the gradient ascent iteration for maximizing a regularized objective. A key difference here is that we can control the weights wk,s to reduce the impact of Du and make the regularization term vanish to zero. Thus the mean μ4 eventually converges to the global maxima.
Proof. Define
Ho == oi)),
Hs = (Hx-1, a
k EK [K-K]] - 1],
μ = E[Ea,1 Hx-1), k E [K].
According to Lemma 2, with freezing C, in class (12), the pre-trained score in Round k is 80x+1 (Iz, t) = (a²()) + h(t)Tp) (x+ - (f))ax) where Fy = E Wk.j and ž, Er Dj [ar].
By choosing yk. and weights wk.j
(Tg) yk M (0" 9k AE.A gh) + gh AAT
TTn, Iz.
Wk.0 1 WW
Wk.j = 0, 1 j < K,
Wk.k = Wkk
where Me > 0, 0
we
will be specified later. And we choose ((1) at Round k as 3(4)
02 Ix-
T ΣΣ
(Io + (t))/((()) 9k-1) Lemma 3 gives the mean of distribution of
2k+1, as
14++1 Tx. + Dge,
(30)
and the output distribution
N FK-1 + Mk- Egk E 5gk-19K- 5
(31)
Applying Lemma 3 yields 24,i E Span (),, thus, Is AAT 3x and E AE,,A AT we get the update rule reduced to
pk+1 AAT ((1 wx)kk) + 3x + 7/. - gh-1
= AA Σ + M) AE,," T fZ - وh (1 - w(--w)AB A AT (Z+ -D))
We set wa
m and 7/-- 77. where A, > will be specified later. Therefore, we have Pk+1 = AAT こΦ + nAΣ,A (hx(Ex),
(32)
24

---

where hx(z) := f(z) (2/2) 1 5 Define J'A argmax,
x- An (x). With some similar steps in proof in Appendix E.2, by choosing BR 2 <-4k 4k (1 c)-2 together with 20 n. we get |KK x =rtnlls- 5* (1ā-=iAml)
MM vd).
w.p. 1- d.
2
2K
with 1)
< 11-11 1In + X) and M1 2 (1+ n22) (1+ log
Also, we can get (28)
+ 2X"
as in proof in Appendix E.2. We restate it here:
X f (*AAN) f(žk) (1=22> - Al15. 11xx - -All33 1)+ L+ || AAA -'AllE- (33)
Since is concave,
f(xA) f (T $ (VI (IA,)) I''s - -x%,2) = =>(E-((AA, -1, - -22A))
Adding (33), holds
f (xa,,) - -((k) - 2 | - 20133-1 - | - 22113-1 -Ilxx - ++7
|| x
2
Due to (32), we have, it holds w.p. - 6,
f (2'A f(uk)= 2 { A|1²² F (L (15* (10-=2uul3-- + +Mid)
We choose À L log K/(4K) and get f (*),N) ((x) x Llog K With assuming | || is bounded, we derive
|ILe p 3 + 1|a - - + Mid. w.p. 1-
K
(FANN) -((x) o (clL2log K
K 8 ) w.p. 1-
log
K
E.4 Auxiliary Lemma The following a standard result in convex optimization utilized in previous proofs.
Lemma 4. Let f be a-strongly concave and B-smooth with respect the (semi) norm I1 for all and y. it holds
a3 I=
(Vf())-Vf())) x a a
a+ [Vf(() v.
(34)
Proof. See Bubeck et al. [9, Lemma 3.11] for a proof.
F
Additional Materials for Experiments
F.1
Simulation
We experiment with our design of the gradient guidance as well as Algorithm and Algorithm 2. Going beyond our theoretical assumptions, we adopt a 15M-parameter U-Net as the score function class for training and fine-tuning our diffusion model.
F.1.1 Experiment Setup
For linear data structure, set the data's 's ambient dimension as D 64 and the linear subspace dimension as d = 16. The linear subspace is represented by an orthogonal matrix A E D We randomly generate a matrix A and fix it once generated. After that, we sample a data point X by first randomly sampling latent variable U NN Ia) and computing X AU. We independently sample a total of 65536 data points as our pre- pre-training data set. For nonlinear data structure, data are uniformly sampled from a unit ball 84
The objective functions considered in our experiments are f1 = 10 - (8T - 3) and fz(z) = 5 0.5|[ | bll. Here, and are randomly generated and fixed afterward. Since our data assumes low-dimensional subspace representation, it is convenient to decompose 0 into 0_ (1 AA )
25

---

and 81 = AATA, representing the off suppor and on-sup components. We refer to ||0,
as the
Tepp
off/on-support ratio. Analogously, for a generated sample, we can also define its off/on- httpoonnsuppottttoo ratio. Clearly, a small oufpon- ratio indicates close vicinity to the subspace.
Score Network Pre-training. We utilize a version of the U-Net [54], with 14.8M trainable parame- ters. Note that this is complicated network going beyond the linear score function class considered in our theories. Following the implementation of Denoising Diffusion Probabilistic Models (DDPM, Ho et al. [34]), we train the U-Net o estimate the score function V log Ptg via minimizing the score matching loss introduced in Eqn. (3). We discretize the backward process have 200 time steps as in Nichol and Dhariwal [47], and the U-Net is trained using our generated data set for 20 epochs. We use Adam as the optimizer, set the batch size as 32, and set the learning rate to be 10-4. After the pre-training phase, we confirmed that the data subspace structure well learned, as the generated samples using the pre-trained diffusion model have average off/on-support ratio of 0.039.
Implementation of Algorithm 1, In each iteration of Algorithm 1, we need to compute the gradient guidance Gloss. We set the targeted y value at the k-th iteration as yı + I 2k. where ō specifies the increment per iteration. The choice on 8 is mnstance-depentee near-optimal in different experiments. For comparing naive gradient with gradient guidance in Figure 8, we set = 0.2 and 0.9, respectively for using naive gradient G and gradient guidance Gloas- In Figure we choose ō to be (a) 0.05, (b) 0.2, (c) and (d) 1, corresponding to each panel. We initialize Algorithm I with a batch of 32 samples generated by the pre-trained model. Each sample determines an optimization trajectory. We repeat Algorithm for 5 times with different random seeds and report the error bars.
and we set it via tuning for
Algorithm 2 differs from Algorithm 1 in that it allows additional fine-tuning of the praitedined score network. In practice, to update the score network incorporating newly generated data, one does not have exactly solve (14) by re-training the full model from scratch. Instead, (14) can be viewed as guideline that motivates more computationally efficient ways for updating the pre-trained score. It is a common practice to only fine-tune the weights of the old model by performing gradient descent over few batches of newly generated data, which is similar to the spirit of (14). To be more specific, strategy: We only perform one Adam optimization step using the re-weighted loss given by Eqn. (14) with batch of 32 generated samples. We set the learning rate as 10-6 This simple strategy already demonstrates good performances shown in Figure 10. Other implementation details are kept the same as those of Algorithm
Implementation of Algorithm 2.
adopt a computationally lightweight fine-tuning
F.1.2 Results
We first demonstrate our gradient guidance Gloss preserves the subspace structure learned from the pre-trained model. For comparison, we also tested the naive guidance G defined following Proposition 3 (with Σ I). For quick reference, repeat the definition here:
G(I,, 1) = 3() (y- [[]r g,
where (() > 0 and y E IR are tuning parameters, and E[zr| is the conditional expectation of xo given noise corrupted data 2,. For implementation, we replace E[zo|z,] by its look-ahead estimator E[rg|ay [ based on the Tweedie's formular.
Comparing G and Gooss on Preserving Subspace Structure. Figure 8 (a), (c) verify that the naive gradient performs much worse than Gloes in preserving the linear subspace structure. It is consistent with our theoretical finding that the gradient guidance Gloss keeps the generated sample close to the latent subspace, with substantially smaller off-support errors. When allowing adaptive score fine-tuning in Algorithm 2, Figure 8 (b), (d) show that the off-support error increases as the model gets fine-tuned using self-generated data, due to increasing distribution shift. Even in this case, the naive gradient G leads to much more severe off- suf-ru errors as compared to Gloss-
Algorithm 1 Converges to Regularized Optima. We plot the convergence of Algorithm in terms of the objective value in Figure 9. Figure 9 (a),(b) are for the objective function fi 10 (ØT x ---33 3)2 as the objective function, while Figure (d) are for the objective fa 5 0.5 æ bl]. We observe
26

---

Predialedd
Pretrained
round
sund
(a) Algorithm
(b) Algorithm 2
(c) 300-350 round of (a) (d) 1000-1200 round of (b)
Figure 8: Comparison between two types of gradient guidance G and G..... We plot the off/on support
ratio of the generated samples, denoted by Toff off/on-support ratio of 9.
| The objective function is fi (z), with 0 having an
Tay
that the algorithm converges reach some sub-optimal objective value, but there remains gap the maximal value. This is consistent with our theory that the pre-trained model essentially acts as regularization in addition to the objective function. Adding gradient guidance alone cannot reach global maxima. This coincides with our theoretical findings Theorem
Gou
Gur
Gus
(a) 0 A3
(b)
||0
(c) b 4 1p
(d) b ~ N/4 LD, 9 Ia)
Figure 9: Convergence Algorithm under different objectives. Objectives are (x)) (a) and (b), and
f2(x)) for (c) and (d). Parameters and are specified as (a) A with 3 being sampled from the unit ball in R (b) the off/on-support ratio of 0 being 9 (same Figure (c) and (d) choosing as homogeneous vector or randomly from a Gaussian distribution. All the experiments adopt gradient guidance Gioss-
Algorithm Converges to Global Optima. Algorithm 2 converges to the maximal value of the objective function fi 10 - (6 x 3) as shown in Figure 10(a). In Figure 10(b), we visualize the distribution of generated samples of Algorithm (blue) and (red), respectively, as the iteration evolves. We see that samples from Algorithm mostly stay close the pre-training data distribution (area described by the dotted contour). In constrast, samples of Algorithm 2 move outside the contour, as the diffusion model gets fine-tuned using self- seff-generated data.
Results for Nonlinear Data Structure. We apply Algorithm to data uniformly sampled from unit ball in R64, The objective reward function is defined as /(x) 0 x, where |[O]| = 1. The left panel of Figure 11 demonstrates that rewards increase and converge when using Algorithm Higher guidance strength A (corresponding to lower regularization) results in higher convergent reward. The right panel of Figure 11 shows that, for the same reward level, gradient guidance achieves smaller deviation from the unit ball compared to the naive gradient approach. This suggests that gradient guidance better preserve data structure for nonlinear manifolds.
F.2 Image Generation
Hyperparameters. Since tuning parameters y and (t) are both for the strength of guidance, can fix one of them. We choose 3()) = 100 as suggested by [71], and set a series of E {2, 10). We run batch size 20 samples parallelly across iterations evaluate the rewards. The prompt uniformly sampled from the 1000 classes of ImageNet [21].
E3 Time Efficiency
We summarize the time cost of our experiments one NVIDIA A100 GPU in Table 1.
27

---

NO-
Gass
(a) Convergence of Algorithm
(b) Distribution of generated samples
Figure 10: Convergence of Algorithm 2. Panel (a) plots the objective values achieved by Algorithm as function of iteration.. Here is chosen the same as in Figure 9 (b) with off/on-support ratio || 9. Panel (b)
To
visualizes the distribution of the generated samples of Algorithm 2 (red) across the iterations. For comparison, we also visualize the distribution of generated samples of Algorithm (bluc).
0117
0.30
gradkem: pudlices
0.00
0.00
03
Figure 11: Nonlinear data structure experiment. We apply Algorithm to data uniformly sampled from a unit ball 61, The objective reward function f(x) = ær where ||0] = 1. Left: Rewards increase and converge with Algorithm 1. Higher guidance strength A (lower regularization) results in higher convergent reward. Right: For the same reward, gradient guidance achieves smaller deviation from the unit ball compared to the naive gradient. This indicates that gradient guidance can preserve data structure for nonlinear manifolds.
Total runtime (iterations)
Per iteration
No guidance
Simulation 3.8 min (50 iter) min (1000 iter) Image
4.6
2.6
1.3 min (5 iter)
15.8
4.9
Table 1: Runtime Efficiency of Algorithm 1. Red refers to the total time to converge. No guidance refers to the time for one-time time inference of the pre-trained model.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The introduction (Section 1) includes paragraph "Scope and Contribution", together with the abstract, clearly reflecting the paper's ' contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: All the theoretical results Lemma 1, Theorem 1, Theorem 2 and Theorem provide the full set of assumptions, and the correspond proofs are in Appendix D.4, Ap- pendix D.5, Appendix E.2 and Appendix E.3, respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The main information of the experiments are described in Section 7, and the details are provided in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in
30

---

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The codebase is released at https//ggitubb.com/yuaay 22CGGMMpMI222
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
ooiicylyyyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.coo wttp:cccpppccccpciia.eeeaec /ttie/gggiddeeeeeeeubbbiiiii SCodeSubmissiiiiiiee.e ssimisolssiiyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The experimental details are provided in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiments that support the main claims of the paper provide error bars, as shown in Figure 4, Figure and Figure 6.
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean,
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All information on the computer resources is included in Appendix F,
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : ////riuuus..p/// cc cc/public/Ethi ctwiissuielliees
Answer:[Yes)
Justification:
The research adheres to all guidelines and principles outlined in the NeurlPS Code of Ethics, including considerations for fairness, accountability, and transparency. Ethical practices were maintained throughout the study, ensuring compliance with all applicable standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
32

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts of this paper Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
33

---

All creators and original owners of assets used in this paper are properly credited. The licenses and terms use are explicitly mentioned and respected according to their respective guidelines.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our code and models along with documentation at http://gghbbbbommm///aaaag//////
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
34

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "fxuvkju0L3", "reviewer": "Reviewer_8pJj", "summary": "This paper rethinks the gradient-based guidance methods through the optimization perspective.  Similar to the manifold assumption, the authors first rely on the assumption that the observed data is from the lower dimensional space. Then, they claim that the naive gradient guidance does not maintain the data subspace, and show how to preserve the data subspace and incorporate guidance by using the look-ahead formulation. For the proof, they rely on the Gaussian linear model. From such motivations from theoretical results, they propose two versions of gradient guidance methods: 1) Only update the gradient guidance components and 2) Simultaneously finetuning the score model. These formulations enable the convergence to the data subspace, unlike naive guidance formulation. Experimental results are conducted on toy examples and image generation settings.", "questions": "- The paper relies heavily on the linear assumption. How reasonable is this assumption in the context of the proposed algorithm? Aren't guidance classifiers and score functions typically more complex than linear functions?\n\n## **Reference**\n- [A] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.\n\n- [B] ELUCIDATING THE DESIGN SPACE OF CLASSIFIER GUIDED DIFFUSION GENERATION, ICLR 2024\n\n- [C] ADJOINTDPM: ADJOINT SENSITIVITY METHOD FOR GRADIENT BACKPROPAGATION OF DIFFUSION PROBABILISTIC MODELS, ICLR 2024.\n\n- [D] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method, Arxiv 2023.\n\n- [E] Loss-guided Diffusion Models for Plug-and-Play Controllable Generation, ICML 2023\n\n- [F] Manifold preserving guided diffusion, Arxiv 2023\n\n- [G] Freedom: Training-free energy guided conditional diffusion model, ICCV 2023.", "limitations": "Linear approximation of the score function in their proof is mentioned as the limitation.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Strong theories are presented in this paper. Rethinking the guidance formulation as the optimization procedure, that aims to optimize 1) guidance and 2) convergence of the data subspace, is interesting.\n- The paper is well-written and easy to follow.", "weaknesses": "- While the proposed algorithm demonstrates considerable theoretical value, its practical implementation appears to be quite slow. This could be perceived as a weakness of the paper. \n- The paper discusses Universal Guidance, yet it omits mention of similar training-free [B, C, D, E, F, G] and training-required guidance [A] mechanisms for utilizing off-the-shelf models. Including a discussion on these mechanisms would enhance the comprehensiveness of the paper. Additionally, it would be beneficial to address the practical and theoretical advantages compared to these mechanisms. As it stands, the paper does not sufficiently clarify these comparative advantages, which leaves some ambiguity about the relative strengths compared to these methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our theoretical value and other comments! We first respond to your main concerns.\n\n>**Weakness 1:** While the proposed algorithm demonstrates considerable theoretical value, its practical implementation appears to be quite slow.\n\n**A1:** During rebuttal, we add a new running time analysis for our experiments, please refer to **Table 1** and the **computational efficiency** section in [our rebuttal][re-link] for more details. \n\n\n>**Weakness 2:** The paper discusses Universal Guidance, yet it omits mention of other similar training-free and training-required guidance mechanisms. Additionally, it would be beneficial to address the practical and theoretical advantages compared to these mechanisms.\n\n\n**A2:** We appreciate your mention of these related works that were omitted, and we've already included them in revision (Sec 4 in [pdf][pdf-link]) and discussed our practical and theoretical advantages compared to them. Here is a summary of the comparison.\n\n- First of all, to adapt a pre-trained diffusion model for guided generation, there are training-required methods [A] and training-free methods, which can be further classified into gradient guidance methods [E,F,G,B] and latent optimization methods [C,D]. Our method falls into gradient guidance methods.\n\n- Generally, classifier guidance [A] requires training/fine-tuning the classifier on noisy input even based on an off-the-shelf classifier, introducing extra training cost than training-free methods. Latent optimization methods are also training free, but are memory-intensive due to the need to store all intermediate gradients when utilizing the chain rule.\n\n- Within gradient-based methods, our work focuses on the theoretical aspect, providing understanding of the guidance method in [E,F,G,B]. Additionally, in our work, we are the first to iteratively apply the gradient guidance as a module for solving optimization problems, and provide theoretical guarantees.\n\nResponse to your other questions:\n\n>**Question 1:** How reasonable is the assumption? Aren't guidance classifiers and score functions typically more complex than linear functions?\n\nA1: Please note that we do not assume the guidance classifier to be linear in our theorems. The form of our guidance is motivated by Gaussian data + linear classifier but its guarantees do not rely on the classifier to be a linear model. Thm 1 also holds for non-linear score, i.e., **arbitrary** distribution. \n\nWe do assume linear score network in Thm 2 and 3, though being restrictive in practice, it is a reasonable assumption for establishing the first optimization theorem for guidance and yielding mathematical insights. The same assumption is also adopted by [38] (Marion et al, 2024), which is the only work we know involving the theory of guided diffusion dynamics. \nIn addition, the effectiveness of our method is beyond linear assumption, justified by our experiments and new simulations (Sec 1 in [pdf][pdf-link] ) added during the rebuttal. More detailed discussion please refer to **Linear Assumption** section in [our rebuttal][re-link].\n\n[re-link]: https://openreview.net/forum?id=X1QeUYBXke&noteId=WsELdf2puF\n[pdf-link]: https://openreview.net/attachment?id=WsELdf2puF&name=pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4eR7DH1Vx", "reviewer": "Reviewer_2yVt", "summary": "Under the assumption of the data belonging to a low-dimensional linear subspace, the authors investigate two common gradient-based guidance techniques of diffusion models, encouraging the use of one of them (computing the gradient at the estimate of x0 given xt, as done in many works).  \nFor concave reward, and under the additional assumption of linear score function, they also establish convergence results for the mean of the guided process.", "questions": "1.  \n\nThe tight connection between gradient-guided diffusion models and proximal optimization algorithms is discussed in detail in:  \nTomer Garber and Tom Tirer, \"Image Restoration by Denoising Diffusion Models with Iteratively Preconditioned Guidance,\" CVPR, 2024.  \nThis relevant literature should be mentioned.\n\n2.  \n\nThe statement in line 48:  \n\"Why does naively guiding diffusion models using gradient never work in practice?\"  \nIs not clear, as there are leading approaches with gradient-based guidance that work, such as (Garber and Tirer, 2024) and the back-projections [53] / least-squares [14] methods that it generalizes.   \nOnly much later in the paper, the reader can see that the authors do not refer to **the common practice** of computing the gradient at the estimate of x0 given xt as a plain gradient-based guidance. This should be stated clearly already at early stage.  \nThis also means that you need to tone down the claim in the contribution list that you: \"introduce guidance based on forward prediction loss\", because this is already a well-known practice.  \n\n3.\n\nBased on your Eq. 1, it seems that you consider the variance preserving (VP) SDE formulation.  \nWhy do you state in line 85 that without loss of generality all your analysis is done for q=1?  \nThis needs to be explained, at least in the appendix, as q=1 is not the common setting (but rather some linearly increase from near 0 to 1).\n\n4.  \n\nThere seems to be some discrepancy between your Eq. 2, associated with Eq. 1, and the SDE formulation of diffusion models [50]. Recall that dt in Eq. 2 is negative.\n\n5.  \n\nThe \"general\" explanation in Section 3.2 that naive gradient-based guidance doesn't work is not convincing because it ignores that fact that the step-size decreases as the t get closer to t=0 and that the noise injection can mitigate error propagation.\n\n6.  \n\nYou state: \"Alg. 1 is light-weighted ... takes 76min overall.\" Recall that the reverse process is not performed in the offline pretraining stage. So obviously Alg. 1 is quite slow. Indeed, it is known that guidance methods that include the DNN's Jacobian computation are slow.\n\n7.  \n\nThe assumptions of the theoretical analyses in the paper are quite strong.  \nEspecially Assumption 1 on the signal belonging to low dimensional linear subspace and the assumption on linear score model (Eq. 12).  \nAlready in the contribution list you should state the assumption on the score model, which essentially leads to convex optimization.\n\n8.  \n\nAny idea how can Theorem 1 be generalized to low dimensional manifold rather than low dimensional linear subspace?  \nUnder the assumptions, in what aspects Theorem 2 differs from standard convex analysis? (maximization of an L-smooth concave function?). \nAren't there any other works that study convergence of guided diffusion models except Marion et al. [38]? There are several works on convergence of diffusion models, e.g., (De Bortoli, 2022). What prevents extending them to the guided case?\n\nValentin De Bortoli, \"Convergence of denoising diffusion models under the manifold hypothesis,\" TMLR, 2022. \n\n9.  \n\nUnder the subspace assumption 1, your data covariance matrix is a DxD matrix of rank d<D.  \nTherefore it is not invertible. I suggest using the dagger symbol to avoid confusion between inverse and pseudoinverse. Provide more details in the proof to clarify that this issue is taken into account.  \nNote that there are some peculiar differences between inverse and pseudoinverse, e.g., generally, pinv(AB) is not equal to pinv(B)pinv(A).\n\n10.  \n\nRegarding Section 5, you need to clearly state what are the differences of Alg 2 from many other works that fine-tune pretrained diffusion models (just google \"adaptive diffusion\" and \"diffusion personalization\").\n\n11.\n\nWhat is the motivation for convergence to the **global** optimum of the guidance objective if one pays in ignoring/omitting the prior information / regularization of the score model? (As seems to be promoted in Sections 5 and 6).  \nThe tradeoff between reward/loss and regularization is trivial and clearly in most case there is some delicate balance between them that a user should look for.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall providing theory for guided diffusion is appreciated.\n\nThe paper formally shows an advantage of a certain guidance over the other.\n\nThe presentation is clear and the paper is convenient to read.", "weaknesses": "The assumptions made in the paper are strong (as discussed below).\n\nA certain weakness of Theorems 2 and 3 is that they consider only convergence of the mean of the procedure do not say anything about the variance.\n\nThere are no new algorithmic ideas in the paper, as the promoted gradient-based guidance is known and common, and also there exist works on fine-tuned/adaptive diffusion models. Thus, the authors needs to tone down some claims in the contribution list and in the abstract.\n\nMore comments are stated below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our effort in providing theory for guidance. We first respond to your main concerns.\n\n>**Weakness 1** Assumptions made are strong. Any idea how can Theorem 1 be generalized to low dimensional manifold rather than low dimensional linear subspace?\n\n**A:** Given the challenge in developing theory for guided diffusion, adopting linear assuptions on data subspace and score funtion should be considered reasonable, more detailed discussion can be found in **Linear Assumption** part of [our rebuttal][re-link]. \n\nTheorem 1 may have the potential to generalize to low-dimensional manifolds. For low-dimensional manifolds, the local geometry around any point can be approximated by its linear tangent space, and current Thm1 shows the guidance lies in this tangent space. By controlling the strength of guidance, the guidance vector on the tangent space will remain close to the manifold.\n\n> **Weakness 1 (Cont.)** In what aspects Thm 2 differ from standard convex analysis? There are  works on convergence of diffusion models, e.g., (De Bortoli, 2022). What prevents extending them to the guided case?\n\n**A:** **Diffusion theory cannot be simply extended to guided diffusion.** The addition of a guidance term disrupts the dynamics of the reverse SDE, thus the output distribution cannot be characterized. Previous theoretical work, such as [17](De Bortoli, 2022), heavily relies on the fact that distribution errors stem from approximation errors in the score function and the initial distribution of SDE. However, the training-free approach doesn't provide an approximation of $\\nabla_{x_t} \\log p_t(y|x_t)$ but instead introduces a plugin, implementation-convenient gradient guidance term. This discrepancy creates a barrier to directly extending existing theories to guided diffusion. [38](Marion et al, 2024) is the only work we know involving the theory of guided diffusion dynamics, and it also requires the linear score assumption. Therefore, this assumption should be considered reasonable. **Our key insight is characterizing the output sample distribution of guided diffusion as taking a proximal gradient step.** We derive the reverse process as a proximal step. This established relationship forms the premise for optimization and distinguishes our analysis from conventional convex optimization. \n\n>**Weakness 2:** A certain weakness of Theorems 2 and 3 is that they do not say anything about the variance.\n\n**A:** This is a misunderstanding, you might have missed the formula we provided for variance in line 235, which points to Eqn 25 in Appx E. This formula demonstrates that the variance of the output distribution is smaller than that of the training data (empirical covariance). Similarly, for Theorem 3, we present the variance in Eqn 31 of Appx E.3.\n\n>**Weakness 3:** There are no new algorithmic ideas in the paper, as the method is common, and also there exist works on fine-tuned/adaptive diffusion models.\n\n**A:**  Algorithmically, we introduce an iterative optimization algorithm that apply the prompted gradient guidance to the local linearization of objective function(Alg 1). This approach is new within the optimization context. In addition, our main focus is more on theoretical understaning for guided diffusion than proposing new method.\n\nResponse to your other questions:\n\n>**Q1:** The statement in line 48 on \"naive gradient guidance\" is not clear. The explanation in Sec 3.2 that naive gradient doesn't work is not convincing because it ignores that fact that the step-size decreases and that the noise injection can mitigate error propagation.\n\n**A1:** We appreciate the reviewer's suggestion, we added a clear definition for \"Naive gradient guidance\" around line 48. To give a more convincing explanation for the failure of naive gradient-based guidance, we add a new lemma showing naive gradient suffers at least a constant error:\n\n**Lemma(Failure of naive guidance)** For naive guidance $\\texttt{G}(X_t^{\\leftarrow}, t) = b(t) \\nabla f(X_t^{\\leftarrow})$, suppose $b(t) >b_0>0$ for $t >t_0.$ For data in subspace under Assumption 1 and reward $f(x)=g^\\top x$, $g \\perp Span(A)$ with $h(t)=1-\\exp(-\\sqrt{t})$, then the orthogonal component of the generated sample is consistently large:\n$$\n    \\mathbb{E} [X_{T,\\perp}^{\\leftarrow}] = C g, \\quad C > \\exp{\\left(-5/2\\right)}b_0.\n$$\n>**Proof.** Under Assumption 1, the score can be decomposed to terms parallel and orthogonal to Span(A) (Prop 2, Appx D.3) Applying naive guidance, we examine the orthogonal reverse process:\n$$\n\\mathrm{d} X_{t, \\perp}^{\\leftarrow} =\\left[\\frac{1}{2}-\\frac{1}{h(T-t)}\\right] X_{t, \\perp}^{\\leftarrow}\\mathrm{d} t + b(t)g \\mathrm{d} t+\\left(I_D-A A^\\top\\right) \\mathrm{d} \\overline{W}_t.\n$$\n\n>Solving this SDE, we get the expectation of the final state following $\\mathbb{E}[X_{T, \\perp}^{\\leftarrow}] = \\int_0^T \\exp \\left(- \\int_0^{t}h^{-1}(s)\\mathrm{d}s \\right) e^{t/2}b(T-t) g \\mathrm{d}t$. For $h(t) = 1 - \\exp(-\\sqrt{t})$, we have the coefficient of direction $g$ is larger than $\\int_0^T \\exp(-t/2-2\\sqrt{t})b(T-t)\\mathrm{d}t > \\int_{0}^1 \\exp(-5/2)b_0 \\mathrm{d}t >0$ where we can assume $T>1$. Thus, $\\mathbb{E}[ X_{T, \\perp}^{\\leftarrow}] \\neq 0$. This means the generated sample is going out of the subspace, i.e., naive gradient guidance will violate the latent structure.\n\n\n\n\n\n[re-link]: https://openreview.net/forum?id=X1QeUYBXke&noteId=WsELdf2puF\n[pdf-link]: https://openreview.net/attachment?id=WsELdf2puF&name=pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reigwpM0pX", "reviewer": "Reviewer_uzJ1", "summary": "This paper proposes a new approach to the problem of gradient-guided generation for diffusion models. The main challenge of gradient-guided generation is to maintain the generated sample within the support of the sample distribution. To address this issue, this work starts with a simplified model, using a linear objective function and a low-dimensional linear sample space, and provides a closed form for the gradient guidance. Under this simplified model, the paper offers a theoretical analysis to show the converged distribution of the generated samples for concave and smooth objective functions. Lastly, it also proposes another variant that allows fine-tuning over pretrained models, further improving optimization performance", "questions": "1. I wonder how the algorithm proposed in this work compares to the direct optimization approaches in [3][4][5][6] (references mentioned above)?\n\n2. Could the author conduct some simple experiments on synthetic data with a nonlinear sample space? I am curious how well the proposed method will perform when the linear assumption is violated.", "limitations": "See my comments above. In summary, the main limitations of this work are:\n\n1. Incomplete literature review and comparison with existing works.\n\n2. The performance of the proposed algorithm appears to be impractical.\n\n3. The presentation can be improved.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of deriving this form of gradient guidance is interesting and novel, particularly the use of a low-dimensional linear sample space to demonstrate the problem of out-of-distribution in gradient-guidance sampling.\n\n2. The theoretical results are solid, and I appreciate the finding that the proposed method can indeed preserve the low-dimensional linear structure theoretically.", "weaknesses": "1. Although I appreciate the motivation and analysis in this work, I do not think this method is practical in reality. Firstly, the core advantage of gradient-guidance approaches like [1][2] is that they do not require gradient backpropagation over the neural network, thus adding only minor computational cost to direct sampling. In contrast, the method proposed in this work requires gradient backpropagation over the neural network, making it prohibitively slow. As mentioned in the paragraph above Algorithm 1, guiding stable diffusion requires 76 minutes for optimization. Secondly, since the method requires gradient backpropagation over the neural network, I think the author should compare it to the serial works on direct optimization in [3][4][5], which are completely ignored in this work. These works target the same task and have the same computational requirements. From my understanding, the algorithm proposed in this work cannot rival direct optimization methods, as the latter do not rely on simplified models to derive the algorithm. In fact, a concurrent work [6] shows that one can optimize SDXL, a 3B parameter model, within 10 minutes, which is in sharp contrast to the performance of Algorithm 1 proposed in this work.\n\n2. The theoretical analysis is certainly novel, but it heavily relies on the linear sample space assumptions. It is unclear whether this method can be applied to cases where the sample space is highly nonlinear. The author conducted a simple experiment with image diffusion, which somewhat investigates this point, but I think the author should also conduct experiments on a highly nonlinear synthetic data distribution to validate if their method works in this scenario.\n\n3. The writing and presentation of the paper can be improved. For example, when moving from Section 3 to Section 4, the target objective function switches from a linear function to a general function without much explanation. I can only guess from Algorithm 1 that the gradient \n$g$ is directly replaced by some stochastic gradient estimator of a general function $f$? I think the author should provide some explanation on this point.\n\n[1] Song, Jiaming, et al. \"Loss-guided diffusion models for plug-and-play controllable generation.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Chung, Hyungjin, et al. \"Diffusion posterior sampling for general noisy inverse problems.\" arXiv preprint arXiv:2209.14687 (2022).\n\n[3] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on\nComputer Vision, pages 7280–7290, 2023b.\n\n[4] Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow:\nDifferentiating through flows for controlled generation. arXiv preprint arXiv:2402.14017, 2024.\n\n[5] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn,\nand Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. arXiv preprint\narXiv:2312.11994, 2023.\n\n[6] Tang, Zhiwei, et al. \"Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization.\" arXiv preprint arXiv:2405.18881 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1:** The proposed method is prohibitively slow due to gradient backpropagation over the neural network. Other approaches like [1][2] do not require gradient backpropagation over the neural network.\n\n**A1:** There seems to be some confusion. Your understanding is incorrect. Both [1] and [2] require backpropagation through the score network. Please refer to:\n\n- In [1], Eqn.2 includes score network, and Eqn.8 takes gradient over it.\n\n- In [2], Eqn.10 includes score network, and Eqn.16 takes gradient over it.\n\nOur algorithm is as efficient as other gradient-based guidance methods while enjoying additional theoretical guarantees. Please see our **new Table 1 in [pdf][pdf-link]** for a detailed analysis of our runtime efficiencies.  \n\n>**Q2:** The method can't rival direct optimization. These works target the same task and have the same computational requirements. \n\n**A2:** Thank you for pointing us to these papers. They are relevant and we have added discussions about these works in our **expanded literature review** in Sec 4 of [pdf][pdf-link]. \n\n- **We target more general tasks and focus on theory.** We respectfully disagree that our method and direct optimization target the same tasks. Indeed they are relevant. However, we consider general optimization over data's latent manifold. Our Section 5 provides the first methodology and theory for adapting diffusion models with self-play and online data collection to **find the global optimum of a user-provided objective function**.\n\n- **They don't have the same computation requirement: Direct optimization has a high memory burden and ours does not.**  While our gradient guidance method enables sampling using $O(1)$ memory, direct optimization methods [3-6] suffer from **O(T)** memory, a substantially higher burden, because they need to backpropagate through the ODE solver, which requires storing all intermediate gradients when utilizing the chain rule. For example, [4] states: *\"backpropagating through the solver can be expensive in memory\"*, similarly, [5] notes: *\"maintaining the intermediate activations for solving the ODE during backpropagation can be memory-intensive. This issue can be addressed with gradient checkpointing or an invertible ODE, at the cost of more computation or model complexity.\"* Thus, our method is much more light-weighted. \n\n>**Q3:** A concurrent work DNO [6] shows that one can optimize SDXL, a 3B parameter model, within 10 minutes, which is in sharp contrast to the performance of Algorithm 1 proposed in this work.\n\n**A3:** Thank you for pointing out this paper. After carefully checking their paper, precise running time was only reported for SD1.5 (Table 1 in DNO[6]), the same model tested in our experiments. For SDXL, there's only a figure with a vague runtime axis, prevents direct comparision. \nWe added a **new Table 1 [our rebuttal][re-link] for analysis of computation efficiency**. For adapting the SD 1.5 model, our method takes 15.8 seconds per optimization round, and <5 iterations to converge (as shown in Figure 6). **Our total runtime is < 2min**, which is comparable to the 10min runtime reported by Table 1 in DNO . \n\n>**Q4:** Could you conduct some simple experiments on synthetic data with a nonlinear sample space?\n\n**A4:** Thanks for the suggestion. We have conducted **additional experiment (Sec 1 in [pdf][pdf-link] )** on data on nonlinear space. Our new result demonstrates that our guidance better preserves the structure of nonlinear manifolds than naive gradient guidance. \n\n>**Q5:** When moving from Section 3 to Section 4, why does the target objective function switch from a linear function to a general function?\n\n**A5:** Yes our task is to generate solutions for general nonlinear, concave optimization problems. Our Section 3 uses the linear model as a motivating example to derive the gradient guidance. Our Sections 4, 5 provide our main results for general nonlinear optimization. \n\nReferences:\n\n[1] Song, Jiaming, et al. \"Loss-guided diffusion models for plug-and-play controllable generation.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Chung, Hyungjin, et al. \"Diffusion posterior sampling for general noisy inverse problems.\" 2022.\n\n[3] End-to-end diffusion latent optimization improves classifier guidance. ICCV, 2023.\n\n[4] D-flow: Differentiating through flows for controlled generation, 2024.\n\n[5] Optimizing diffusion noise can serve as universal motion priors, 2023.\n\n[6] Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization. arXiv preprint, 2024.\n\n\n[re-link]: https://openreview.net/forum?id=X1QeUYBXke&noteId=WsELdf2puF\n[pdf-link]: https://openreview.net/attachment?id=WsELdf2puF&name=pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0IF2E7yh1", "reviewer": "Reviewer_HWVj", "summary": "This paper investigate gradient guidance for adapting or fine-tuning pre-trained diffusion models from an optimization perspective.\nThe author proposed a look-ahead loss based gradient guidance and two variants of diffusion-based generative optimization algorithms utilizing it.\nThe author provided theoretical guarantees for adapting/fine-tuning diffusion models.", "questions": "No questions", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, although this is a theory paper, it is very impressive. It used theoretical approaches to explain some problems of diffusion with guidance:\n1. naive guidance doesn't work on latent structure of data\n2.  algorithm on guided diffusion using gradient queries on new samples\n3. proposed an adaptive gradient guided diffusion, where both pre-trained score network and guidance are iteratively updated using self-generated samples.", "weaknesses": "No obvious weakness.\nIf possible, please include the reason for each step in your proof, i.e. the first step follows from???, the second step follows from???. Then it will be a perfect paper. I will increase the score if you can fix this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our theoretical contributions and your valuable suggestions! We've added more detailed explanations of the derivations in the proof.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for their valuable feedback! 

### **New Theory, Experiments, and Running-Time Analysis**

**Theory for the failure of Naive gradient:** We construct a rigorous counterexample Sec 3 in [pdf][pdf-link] showing that the generated samples will suffer at least a constant error if the gradient of reward is directly applied as guidance.

**Experiment with data from nonlinear manifold:** We test our guidance on data from nonlinear manifold Sec 1 in [pdf][pdf-link] and plot the off-support deviation and reward curve. The curve shows that our guidance better preserves the data manifold than naive gradient guidance. 

**Break-down analysis for running time:** We report and analyze the running time of our algorithms:

| | Total runtime (iterations)  | Per iteration | No guidance |
|------------|:-----------:|:-----------:|:-----------:|
| Simulation | **3.8 min** (50 iter),  76 min (1000 iter) | 4.6 s | 2.6 s |
| Image | **1.3 min**  (5 iter) |  15.8 s | 4.9 s |

**Table 1: Runtime Efficiency of Algorithm 1.** **Bold** refers to the total time to converge. No guidance refers to the time for one-time inference of the pre-trained model.

### **Expanded literature review**
We appreciate reviewers pointing us to several papers that are not directly under the topic of "diffusion model for optimization" but yet highly related. We include those in our revised literature review section in [pdf][pdf-link]. Gradient-based guidance and direct latent optimization are two main routes for adapting a pre-trained diffusion model to some reward, in a training-free way.  Our method falls into the class of gradient-based guidance, so we recap the comparison to other methods in this class here:

Gradient-based Guidance [A,B,C,D,E] utilize gradient as guidance. [A,B,D] solves inverse problem image and [C, E] solves guided/conditional image generation. [A,C] both propose similar guidance: on the predicted clean data $x_0$ with respect to $x_t$. Differently, our paper provides the first rigorous theoretical study of this gradient-based guidance from an optimization perspective, add prepose the first iterative algorithm that enjoys provable convergence guarantee.


### **Clarification on Our Contribution and Scope**

We want to highlight that our main contribution is to **provide the first optimization guarantee for adapting a pre-trained diffusion model to task-specific needs via gradient guidance**. Specifically: 1) We demonstrated the gradient guidance preserves learnt data structure (Thm 1); 2) We established optimization convergence theory for the gradient-guided diffusion model and its iterative fine-tuned variant (Thm2, 3; Fig 1).

### **Discussion on Linear Assumptions**
- **Linear case is a fundamental setting for theory and is highly nontrivial for theory on guided diffusion.** Extending diffusion theory to guided diffusion is highly nontrivial. The guidance term disrupts the reverse SDE dynamics which previous works in diffusion theory, such as [17] (De Bortoli, 2022), heavily rely on, making it impossible to characterize the output distribution if without further assumptions. [38] (Marion et al, 2024) is the only work we know involving the theory of guided diffusion dynamics, and it also requires the linear score assumption. Therefore, this assumption should be considered reasonable.

- **Practical usage and satisfying properties of our guidance and algorithms extend beyond linear assumption.** 
    - **For assumption on linear score network:** When deploying, guidance *Gloss* is easily constructed for **any** pre-trained diffusion network, not limited to linear networks. Satisfying optimization performance was validated by experiments (Sec 6.1) with a 15M-parameter U-Net score network. 
    - **For linear assumption on data manifold:** Theoretically, *Gloss* always preserves the data subspace for **arbitrary** latent distribution, not limited to Gaussians (Lem 1). For nonlinear data manifolds, our experiment (Sec 6.2) generated highly human-interpretable images, suggesting the manifold preserving property of *Gloss* stays for highly nonlinear manifolds. The new simulation in Sec 1 of [pdf][pdf-link] on sphere subspace also verified this property extending beyond linear assumption.

### **Clarification on Computation Efficiency**
Some reviewer misinterpret our discussions on computation efficiency that 
> a single run of the backward sampling process (Module 1) takes 4.6s, and Alg. 1 takes 76min overall.

Here "76 min" is the total time for running to 1000 optimization rounds (one round corresponds to one complete reverse process with diffusion model), averaging 4.6 seconds per round in our simulations. Most of our simulations converge within 50 rounds, thus runs only 3.8 min to get satisfying results. 

For Stable Diffusion v1.5, our approach needs 15.8 seconds per round. Typically, it achieves satisfactory results in fewer than 5 rounds (Figure 6), totaling 1.3 min. Reads are summarized in the **Table 1**.

**Backpropagation gradient through denoiser has been adopted by multiple gradient guidance method, thus it is NOT a deal breaker for efficiency.** Gradient-based guidance methods like [A,B,C] all require similar gradient backpropagation over the score network, thus our approach is not more computationally expensive in this regard.

References:

[A] Diffusion posterior sampling for general noisy inverse problems, 2022

[B] Loss-guided diffusion models for plug-and-play controllable generation, 2023

[C] Freedom: Training-free energy-guided conditional diffusion model. ICCV, 2023.

[D] Manifold preserving guided diffusion. arXiv preprint arXiv:2311.16424, 2023

[E] Towards accurate guided diffusion sampling through symplectic adjoint method. arXiv preprint arXiv:2312.12030, 2023b

[pdf-link]: https://openreview.net/attachment?id=WsELdf2puF&name=pdf

Meta-review: The paper provides new theoretical and algorithmic insights into gradient guidance of diffusion models, which is the practice of adding an offset vector to the pre-trained score function estimate in the reverse SDE of a diffusion model with the goal of controlling the output toward a user-specified objective. The contributions are primarily theoretical and deal with guidance based on the “look-ahead” loss for estimating x0 given xt. The theory connects this type of gradient guidance to optimizing a user-specified loss function over the data subspace under the assumption that data lies on a linear subspace. The main convergence results are developed under the additional assumption that the pre-trained score-function estimator is a linear function of xt. 

Reviewers unanimously appreciated the theory, calling it “interesting and novel”, “solid”, “very impressive”, etc.; Reviewer 2yVt appreciated that it provided theoretical evidence favoring one type of guidance over another. The primary concerns were about: (1) strong assumptions underlying the theoretical results, (2) efficiency in practice, (3) the tone of some claims in the introduction being too strong given the assumptions and/or prior evidence. The efficiency concerns were adequately addressed during the rebuttal. The assumptions and their implications were also discussed and some clarifications made. The meta-reviewer notes that assumptions were all clearly stated in the methods section, and all reviewers found the theory interesting and relevant despite not being completely general. I believe the issue is the matching of the claims in the into to the evidence provided by the theorems. I urge the authors to follow the suggestion of Reviewer 2yVt to temper the optimism surrounding the convergence results in the introduction by stating clearly in the introduction the assumption of a linear score network. I also echo the advice to more clearly reference prior evidence surrounding the use of “look-ahead” vs “naive” gradient guidance. With that said, it is clear that even with slightly toned down claims the paper provides valuable theoretical understanding for a fast-moving area.
Decision: Accept (poster)