Variational Control for Guidance in Diffusion Models
Authors: Kushagra Pandey
Farrin Marouf Sofian
Felix Draxler
Theofanis Karaletsos
Stephan Mandt
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
Inverse Problems
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a plug and play method for guidance in diffusion models using optimal control

Abstract: Diffusion models exhibit excellent sample quality, but existing guidance methods often require additional model training or are limited to specific tasks. We revisit guidance in diffusion models from the perspective of variational inference and control, introducing \emph{Diffusion Trajectory Matching (DTM)} that enables guiding pretrained diffusion trajectories to satisfy a terminal cost. DTM unifies a broad class of guidance methods and enables novel instantiations. We introduce a new method within this framework that achieves state-of-the-art results on several linear, non-linear, and blind inverse problems without requiring additional model training or specificity to pixel or latent space diffusion models. Our code will be available at https://github.com/czi-ai/oc-guidance.

Variational Control for Guidance in Diffusion Models
Kushagra Pandey"
Farrin Marouf Sofian 1
Felix Draxler'
Theofanis Karaletsos Stephan Mandt 13
Abstract
A trained diffusion model can be viewed as a steerable stochastic system that follows learned dynamics. By mod- ifying its inputs, we can guide it in the spirit of stochastic optimal control - aiming to reach desired terminal states (e.g., matching guidance signal) while staying close to its native trajectory distribution. This intuition motivates our formulation of classifier guidance as a variational control problem (Kappen, 2008). Inspired by Control as Inference frameworks (Kappen et al., 2012; Levine, 2018), we model guided diffusion controlled Markov process, where control signals are treated as variational parameters. We then apply variational inference to optimize these controls, ensuring that generated samples satisfy terminal constraints without straying far from the unconditional sample mani- fold (see Fig. la). We refer to this framework as Diffusion Trajectory Matching (DTM).
Diffusion models exhibit excellent sample qual- ity, but existing guidance methods often require additional model training or are limited to spe- cific tasks. We revisit guidance in diffusion models from the perspective of variational infer- ence and control, introducing Diffission Trajec- tory Matching (DTM) that enables guiding pre- trained diffusion trajectories to satisfy termi- nal cost. DTM unifies broad class of guid- ance methods and enables novel instantiations. We introduce a new method within this frame- work that achieves state-of-the-an results on sev- eral linear, non- non-linear, and blind inverse prob- lems without requiring additional model training specificity to pixel or latent space diffusion models. Our code will be available at https: //github. com/ czi - ai /oc- wwroooo---oiiiieeeeeeeeeeeeeeee- dance.
Recent work on steering diffusion models has already incor- porated ideas from optimal control (Huang et al., 2024; Rout et al., 2025). However, these works focus on restricted class of control problems. This obscures the available de- sign choices revealed through our novel framework. Indeed, we find that DTM generalizes and explicitly contains a large class of prior work on guidance. We demonstrate the util- ity of this generalization by introducing a new sampling algorithm that seamlessly integrates with state-of-the-art diffusion model samplers like DDIM (Song et al., 2021a) and adapts well to diverse downstream tasks.
1. Introduction
Diffusion models -Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020) and related families (Lipman et al., 2023; Albergo & Vanden- Eijjden, 2023; Liu et al., 2023) exhibit excellent synthesis quality in large-scale generative modeling applications. Additionally, due their principled design, these models exhibit great potential in serving as powerful generative priors for downstream tasks (Daras et al., 2024).
To summarize, we make the following contributions:
Consequently, guidance in diffusion models has received significant interest. However, the dominant approaches to classifier guidance (Dhariwal & Nichol, 2021) and classifier- free guidance (Ho& & Salimans, 2021) require training addi- tional models or retraining diffusion models for each condi- tioning task at hand, are based on simplistic assumptions detrimental to sample quality (Kawar et al., 2022; Chung et al., 2022a; Song et al., 2023; Pandey et al., 2024b).
We propose Diffusion Trajectory Matching (DTM), a generalized framework for training-free guidance based on variational control perspective. DTM sub- sumes many existing and novel guidance methods.
We instantiate our framework Non- lon-linear Diffussin Trajectory Matching (NDTM), which can be readily integrated with samplers like DDIM.
"Equal contribution 'Dpartment of Computer Science Univer- sity of California, Irvine "Chan-Zuckerber Initiative 'Department of Statistics, University of California, Irvine. Correspondence to: Kushagra Pandey <pandeykl @ uci.edu>.
We show that NDTM outperforms previous state-of- the-art baselines for solving challenging and diverse inverse problems with pretrained pixel-space (see Fig. 1) diffusion models. Furthermore, NDTM can be ex- tended trivially to latent-space diffusion models like Stable Diffusion (Rombach et al., 2022) for tasks like style guidance (see Fig. 3).
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Variational Control for Guidance in Diffusion Models
2. Background
approximations of the diffusion posterior. We will show in Section 3.4 that elassifier guidance in diffusion models is special case of our proposed framework. Next, we discuss our proposed framework, which generalizes to diverse tasks like inverse problems and style-guided generation without specificity to pixel or latent-space diffusion models.
Diffusion Models. Given a perturbation kernel p(x+|xx) = N Wp, of La), diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) invert the noising process by learning corresponding reverse process parameterized as,
q xxT) = II (xx=)x)))
(1)
3. Guidance with Diffusion Trajectory Matching (DTM)
The reverse diffusion posterior is specified as ((xx-1(x,) N (xx, 1), Ia) where (...).) is learned via score matching (Hyvärinen & Dayan, 2005; Vincent, 2011; Song & Ermon, 2019). Analogously continuous-time diffusion models (Song al., 2020; Karras et al., 2022) assume that forward process
We now propose a novel framework based on variational control for guidance in diffusion models. Our framework can be directly applied to pretrained diffusion models with- out requiring model retraining. For the remainder of our discussion, we restrict our attention to diffusion models and discuss an extension to flo-matching (Lipman et al., 2023) in Appendix A.3.
dx,= f())x, dt + g(t) dw,, t [0,
(2)
with drift f(1) and diffusion coefficients g() and stan- dard Wiener process W,, converts data Xo Rd into noise
In the following, we first formulate guidance in discrete-time diffusion models as a variational optimal control problem (Section 3.1) following Kappen et al. (2012), which we refer to Diffusion Trajectory Matching (DTM). We then present specific parameterizations of DTM in Section 3.2, which we work out as a guidance algorithm in Section 3.3. Lastly, in Appendix A.2, we transfer the DTM framework to continuous- diffusion models (Song et al., 2020) and recover prior work in guidance in diffusion models.
XT. A reverse SDE specifies how data is generated from noise (Anderson, 1982; Song et al., 2020),
dx,- = [f(t)x+ - vxt) logp:(x.)) dt 9() (3)
which involves the score Vx, (x.) of the marginal distribution over X2 at time t. The score is intractable to compute and is approximated using a parametric estimator (x(x,, t), trained using denoising score matching.
3.1. Variational Control for Diffusion Guidance
Classifier Guidance in Diffusion Models. Given pre- trained diffusion model se (x t), it is often desirable to guide the diffusion process conditioned on input y. Conse- quently, the conditional diffusion dynamics read
The idea of our guidance framework is to take a controlled deviation from the unguided diffusion trajectory implied by Eq. (1) in Section which we repeat for convenience:
Q ((xxTT-i(xT)) = II q(xx-1xx))
(6)
dx,- ((())x-9(tt2²A. V log p(x:) y) dt (())) (4)
To steer the trajectory towards a target state fulfilling exter- nal constraints, we introduce a control signal u, at every time t. This yields the following guided dynamics for given initial state xy:
In classifier guidance (Dhariwal & Nichol, 2021), the condi- tional score can be decomposed as
Vx, x, log p(x:(()) = so (xe, t) p. Vx, logp(y/x:).
(5)
P p(xoo UIT) = II [[x(x--le uz).
where Pe is the guidance weight. The noisy likelihood score is often estimated by training noise-conditioned estimator. is also common to estimate this likelihood via p(y(xx, = p(xo x; ())))()))dxs For example. Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) approxi- mates the diffusion posterior as, (xx))x)) = 8 8(E[xo) xx)), where |[xx] is Tweedie's 's estimate of the posterior at x; (Efron, 2011) 1). This approximation of the diffusion pos- terior in DPS results in a high sampling budget and high sensitivity to the gradient weight Pt- More expressive ap- proximations (Song et al., 2023; Pandey et al., 2024b) result in specificity to linear inverse problems and thus cannot be extended to latent-space diffusion models trivially. We refer to Daras et al. (2024) for an in in-depth discussion on explicit
While model the guided dynamics Markovian due to convenience, non-Markovian approximations are also possi- ble (Li et al., 2021). Given a set of external constraints, the task is to determine the variational control u, Consequently, following Kappen et al. (2012), we can pose this problem as a stochastic optimal control problem with the terminal and transient costs formulated as,
C(XT, u.:T) WT Exprp (xx)) DKI. (P 11)
(8)
Terminal Cost Cs
Transient Cost
The terminal cost in Eq. (8) encodes desirable constraints
on the final guided state while the transient cost ensures
2

---

Variational Control for Guidance in Diffusion Models
Method
Non-Linear Deblurring
Reference
Degradation
NDTM (Ours)
RED-diff
DPS
Materenes
Degratel Iaange
+7
NDIM (0un)
DPS)
p(sv)
p(Kal))
DPS
Imagenet
(8)
Figure 1. Our method guides diffusion sampling to fulfill external constraints. this end, we optimize local direction uz via external constraints while respecting the original trajectory, Eq. (11) (left, center). This recovers accurate reconstructions across tasks compared to classical guidance methods: Nonlinear deblurring (Right). Our method accurately captures most details, while competing methods introduce artifacts in the generated reconstructions.
that the guided trajectory does not deviate strongly from the unguided trajectory, that the final guided state Xo lies near the image manifold. The two losses are traded by scalar WT.
principled, but is difficult to jointly optimize for all controls U1:7 due to the need backpropagate through the entire diffusion trajectory. To avoid this computational overhead, we make several simplifications that make the objective computationally tractable. We justify the validity of the modifications through our empirical results Section 5.
Choice of Terminal Cost. The terminal cost in Eq. (8) en- codes desirable constraints on the final guided state. For in- stance, (xx) x bogg(y]Xo could be the log-likelihood of probabilistic classifier for class-conditional generation or the degradation process for solving inverse problems (see Section for exact form of terminal costs for different tasks). For instance, Huang et al. (2024) adapts guidance for a non-differentiable terminal cost using a path integral control (Kappen, 2005) formulation. While an interesting direction for further work, we only assume that the terminal cost differentiable for now.
First, we optimize u, in a greedy manner, that is at any time in the diffusion process we optimize ut assuming that the remaining steps t 1,...; are unguided. After optimizing for u, we sample from the optimized posterior X p( x, [x, uz) and iterate. When the variational control of u, is flexible enough, suboptimal greedy choices early in the trajectory can be compensated for later.
Second, we evaluate the terminal cost at the current expected final guided state via Tweedie's Formula for E[xo (xe, ur]:
Choice of Divergence. We use the KL-Divergence it decomposes over individual timesteps,
Go Ex, 1 (xx)) x (E x, uzl) +xx))
(10)
where have approximated p(xo Xx, ur) 8(x0
x6).
Bx,, Cr = Σ Ex, (DRI(p(xxx u:) ((xx-1(xx))] (9)
Diffusion Trajectory Matching (DTM). Together, the op- timization problem to solve at time given a position x, reads:
Note that other divergence measures can be useful depend- ing on the specific form of the diffusion posterior (Nachmani
C(u.) = w7(())) + DKL ue) (xx)-1(x))).
et al., 2021; Zhou et al., 2023; Pandey et al., 2025; Holderri- eth et al., 2025).
(11)
We refer to Eq. (11) as Diffusion Trajectory Matching (DTM).
Simplifications. The proposed loss in Eq. (8) generic and
3

---

Variational Control for Guidance in Diffusion Models
Algorithm 1 NDTM (DDIM). Sampling proceeds by infer- ring the control (shaded) followed by sampling from the guided posterior (shaded) at any time
From a practical standpoint, since unconditional score models usually parameterized using neural networks with an input noisy state and a timestep embedding, we further parameterize the posterior mean [x, u,. 1o (X4, u, 1), 1), where the aggregation function f R x Rd x IR + IR combines the noisy state x, and the control u, appropriately.
1: Input:
Optimization Steps: N, Guidance: 7, Pretrained
denoiser: co(..-), Timestep schedule: {1]==o- DDIM Coefficients: 0₁ Dr, Loss Weights: Te. Ki, WT
2: Initialization: XT ~ (V Ia)
this work, we choose an addi- tive form of f x, yuu where y is the guidance weight used to update the current noisy state X; in the direction of the control signal u, We leave exploring other aggregation functions as future work. Moreover, in practice, we sample from a single diffusion trajectory and therefore omit the expectation in Eq. (11). Consequently, the control cost in Eq. (11) can be simplified as,
3: for mort= T to do
4: u, =0
5: Euncond f (x( t)
6: for i 0 to N do
commall (x( + yu, (1) .t) x0 + E(x, x,, + +u, 1
(1)
7:
8:
Cruoe - (uuncooed control
C(u.) = || (x: + yus, () po 1)122 + wr (x)).
10:
Countol = x² x221uul11
(16)
Cerminal = 2T (1)
11:
Due to the non-linear dependence the guided posterior on the control signal uy we refer to the transient cost specifica- tion in Eq. 16 Non-Linear Diffusion Trajectory Matching (NDTM). We will show in Section 3.4 that linear control can be formulated as a special case of this parameterization, yielding classifier guidance. Next, we instantiate the NDTM objective practically.
cl + Canoee + Coontrll Germinal
12:
(1+1) +- Update(u, (5) V., c(())
13: ut
14: end for
15: X,-1 t (DIM(x; ( + +,
16:
end for return Xo
17:
3.3. Specitic Instantiations
Here, we present a simplified form of the NDTM objective in the context of DDIM (Song et al., 2021a).
Continuous-Time Variants. To apply DTM to continuous- time diffusion and flow matching, we adapt the transient costs C.. We call the following Continuous-Tine Diffusion Trajectory Matching, derived for continuous-time diffusion (Song et al., 2020) in Appendix A.2:
Proposition 3.1. For the diffusion posterior parameteriza- tion in DDIM (Song et al., 2021a), the NDTM objective in Eq. 16 has the following tractable upper bound (see proof in Appendix A.1),
(x(; Cr g(1)2 Ex, [[s)((x,)))) I) sa (x: un 011]]
(12)
C(u.) S N2 Uu 1 ()),
1)1l W ((
(17)
Similarly, for flow matching (Lipman et al., 2023; Liu al., 2023; Albergo et al., 2023), refer to this as Flow Trajec- tory Matching (FTM), see Appendix A.3:
where x; = Xt + ue is the guided state and the coefficients
" Vor and Tt /I ac-1 -07 are time-dependent scalars.
Van-il (1--)))1)))
you
Cu (x,, t) (x uz 2.
(13)
The coefficients a and or are specific to DDIM (see Ap- pendix A.1 for more details). Intuitively, the simplified NDTM loss in Eq. (17) measures the deviation between the guided and unguided dynamics, penalizing the magni- tude of the control signal u, (first term) and deviations in the noise predictions (second term). On the contrary, the terminal loss ensures that the expected final guided state satisfies the external constraints. Therefore, the first two terms in Eq. (17) act as regularizers on the control signal uy In Appendix A.2, we derive this simplification also for continuous- time diffusion models. Lastly, worth noting that the control loss in Eq. 17 can be generalized as,
Next, we present a specific parameterization of our frame- work and its instantiation using standard diffusion models.
3.2. Non-linear Diffusion Trajectory Matching (NDTM)
In the context of Gaussian diffusion models (Ho et al., 2020:
Song et al., 2020; Karras et al., 2022), the unguided diffusion posterior is often parameterized as,
q(x-1|xx) (Ho t), t), of
(14)
In analogy to how the unguided diffusion denoising process q is parameterized, we define our controlled process p as
C(u)) = we w//lu, 11 w. wellso(Xe, t) ca cx t) 1|2 wy 4 (xx)) (18)
p(x;-1 X,, nz) = (H u, 1), of Ia).
(15)

---

Variational Control for Guidance in Diffusion Models
In this work, unless specified otherwise, we set w, = -ē. We xÿ. However, exploring alternative weighting sched- ules for different loss terms can be an interesting direction for further work.
Here, we focus on training-free guidance during inference. In this context, there has been some recent progress in ap- proximating the noisy likelihood score (see Eq. 5) by ap- proximating the diffusion posterior pxxo|>)). For instance, DPS (Chung et al., 2022a) approximates the diffusion poste- rior by a Dirac distribution centered on Tweedie's 's estimate (Efron, 2011). This has the advantage that the guidance can be adapted linear and non-linear tasks alike. However, due to crude approximation, DPS converges very slowly and, in our observation, could be unstable for certain tasks (see Table 1). Consequently, some recent work (Yu et al., 2023; Bansal et al., 2024) adds a correction term on top of the DPS update rule to better satisfy the constraints. MPGD (He et al., 2024) attempts to alleviate some of these issues by leveraging the manifold hypothesis. In contrast, our pro- posed method instead directly estimates the guided posterior at each sampling step, thus sidestepping the limitations of DPS in the first place.
Putting it all together. To summarize, at each diffusion time step I, we estimate the control signal u, by minimizing the cost C(u,) (for instance Eq. (17) for DDIM). This iter- ative optimization allows the model to dynamically adjust the control to best align the trajectory with the desired ter- minal cost while minimizing deviations with the unguided diffusion trajectory. Finally, we sample from the guided posterior p(x;-1/X4 Xx, u²). We provide a visual illustration of the NDTM algorithm in Fig. la and its pseudocode implementation in Algorithm
3.4. Connection to Existing Guidance Mechanisms
In this section, we rigorously establish a connection between optimal control and classifier guidance: Our variational formulation in Section 3.1 captures existing approaches. We derive this result in the continuous- variant, as this allows for a closed-form solution of the control problem.
It is worth noting that our method resembles DCPS (Janati et al., 2024), which adopts a similar approach of learning a series of potentials to sample from the final posterior. How- ever, DCPS involves an additional overhead of Langevin Monte Carlo sampling in addition to the posterior optimiza- tion step. Morcover, the control perspective adopted in this work helps contextualize prior work in guidance within our framework (see Section 3.4), which is lacking in DCPS. Some recent methods like TFG (Ye et al., 2024) also pro- pose general frameworks for guidance in diffusion models. However, our proposed method does not fit within their framework.
In particular, let us choose a linear parameterization of the guided score in Eq. (12), that is so (X,, ue, t) = (x(,,, t) + uy. Then, the transient cost reduces to:
Ci Gr= f ||
||
(19)
This is exactly the case of the well-established Path Integral Control (Kappen, 2005; 2008). The solution of this optimal control problem in Eq. (8) reads (Kappen, 2008, Eq. (34)):
More recent work (Kawar et al., 2022; Wang et al., 2023; Song et al., 2023; Pandey et al., 2024b; Pokle et al., 2024; Boys et al., 2024) relies on expressive approximations of the diffusion posterior. While this can result in accurate guidance and faster sampling, a large proportion of these methods are limited to linear inverse problems, which fur- ther limits their application to pixel space diffusion models. In contrast, our method can be adapted to generic inverse problems and is thus agnostic to the diffusion model archi- tecture. Lastly, another line of work in inverse problems approximates the data posterior p(xply) using variational inference (Blei et al., 2017: Zhang et al., 2018a). For in- stance, RED-diff (Mardani al., 2024) proposes learn an unimodal approximation to the data posterior by leveraging diffusion prior. However, this can be too restrictive in practice and comes at the expense of blurry samples. We refer interested readers to Daras et al. (2024) for a more detailed review of training- free methods for solving inverse problems in diffusion models.
(20) Notably, if the terminal cost takes the form of classi- fier likelihood (xx) xx - -log it can be shown (Huang et al., 2024) that the optimal control simplifies to classifier guidance (Dhariwal & Nichol, 2021): of = v(t)wx)x,,(((xxx). This puts a large class of methods approximating the expectation over the posterior p(xp) xx) (Chung et al., 2022a; Song et al., 2023; Pandey et al., 2024b; Huang et al., 2024) into perspective: In terms of our DTM framework, they perform optimal control with linear con- trol mechanism (i.e. control added linearly to the score function). Empirically, we will see in Section 5 that our generalization to non-linear control provides significant per- formance improvements.
u = g())wT Vx, log (expm//xe)exxp(((xxx))))))))
4. Related Work
Conditional Diffusion Models. In general, the conditional score Vx, p(xpp)))))))))))))))))))())))p needed for guided sampling can be learned during training (Saharia et al., 2022; Podell et al., 2024; Rombach et al., 2022) or approximated during infer- ence (see Daras et al. (2024) for a detailed review).
Optimal Control for Diffusion Models. There has been some recent interest in exploring connections between stochastic optimal control and diffusson models (Berner
5

---

Variational Control for Guidance in Diffusion Models
al., 2024). Chen et al. (2024) leverages ideas from control theory for designing efficient diffusion models with straight- line trajectories in augmented spaces (Pandey & Mandt, 2023). Since our guided sampler can be used with any pre- trained diffusion models, our approach is complementary to this line of work. More recently, SCG (Huang et al.,
Table I. Comparisons on noisy Non-linear Deblur. NDTM out-
performs competing baselines by significant margin. Bold: best.
FFHQ (256 256)
ImageNet (256 256)
Method
LPIPS.
FID
KID 4
LPIPS|
FID 4
KID 4
DPS
0.752
249.01
0.139
0.888
346,82
0.2186
2024) leverages ideas from path integral control to design guidance schemes with non-iifferentiable constraints. In contrast, we only focus on differentiable terminal costs, and extending our framework to non-differentiable costs could be an important direction for future work. Lastly, Rout et al. (2025) propose CB-Modulation, a method based on control theory for personalization using diffusion models. Interest- ingly, while RB-Modulation is primarily inspired by a class of tractable problems in control theory, it is a special case of our framework in the limit of w. O, we 0 and 7 = Therefore, our proposed framework is more flexible
RED-diff
0.362
64.57
0.036
0.416
78.07
0.0224
MPGD
0.636
113.98
0.086
0.832
148.96
0.085
RB-Modulation
0.064
19.92
0.0032
0.249
47.60
0.0078
NDTM (ours)
0.046
14.198
0.0004
0.163
34.31
0.0032
and ImageNet (256 x 256) (Deng et al., 2009) datasets, us- ing a held-out validation set of 1, 000 samples from each. For FFHQ, we use the pre-trained model provided by Chung et al. (2022a), and for ImageNet, we use the unconditional pre-trained checkpoint from OpenAI (Dhariwal & Nichol, 2021). For style guidance, following MPGD (He et al., 2024), we randomly generate Ik (prompt, image) pairs us- ing images from WikiArt (Saleh & Elgammal, 2015) and prompts from PartiPrompt (Yu et al., 2022). We use the pre-trained CLIP encoder and Stable Diffusion 1.4 models as the feature extractor F and diffusion model, respectively.
5. Experiments
While our method serves as a general framework for guid- ance in diffusion models, here, we focus on solving inverse problems and style-guided generation. Through both quan- titative and qualitative results, we demonstrate that our ap- proach outperforms recent state-of-the-ar baselines across these tasks using pretrained diffusion models. Lastly, we emphasize key design parameters of our proposed method as ablations. We defer all implementation details to App. B.
Tasks and Metries: dom inpainting, super-resolution, and non-linear deblurring. Additionally, we consider blind image deblurring (BID) task where we additionally infer the deblurring kemnll along with the final reconstruction. We set the noise level By 0.01 for all inverse problems.
For inverse problems, we consider ran-
Problem Setup and Terminal Costs: For inverse problems, given a corruption model A and noisy measurement y E Rd the goal is to recover the unknown sample Xa ~ Pdata- from the degradation y = A(zo) + oyZ, z ~~ ~ (0, Ia). For linear inverse problems, y = Azp. In the case where only the functional form of the degradation operator A is known but its parameters are not, the problem is known as Blind Inverse Problem. For inverse problems, we consider the following form of the terminal cost & (xb) in Eq. 17,
For quantitative evaluation on inverse problems, we report metrics optimized for perceptual quality, including Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018b), Fréchet Inception Distance (FID) (Heusel et al., 2017), and Kernel Inception Distance (KID) (Binkowssi et al., 2018). For completeness, recovery metries like the Peak Sognal-to-Noise Ratio (PSNR) are provided in Ap- pendix C. With the exception of BID (for which we use 100 images), we evaluate all other inverse problems on Ik images. For style guidance, following prior work Yu et al. (2023); He et al. (2024), we report the CLIP score (which measures prompt adherence) and the Style Score (which measures style adherence) on Ik text and image pairs.
* (xx)) = 1l 1
(21)
where xi is the Tweedies estimate at any given time t. We also consider the task of style guidance with Stable Diffu- sion, where the goal is to generate samples adhering to specific prompt and reference style image. More specifi- cally, given a reference style image pretrained feature extractor (like CLIP) F, a pretrained decoder 'D such that x11 = D(2)), we define (xx)) as,
5.1. Inverse Problems
We first evaluate the proposed NDTM sampler against com- peting baselines for non-linear, blind, and linear inverse problems. We provide all hyperparameter details for our method and competing baselines in Appendix B.
क Φ($5)= (g(F(()) - ((
(22)
where G denotes the Gram-matrix operation and ||-||F de- notes the Frobenius norm. Note that prompt adherence is achieved via the pretrained Stable Diffusion model.
Non-Linear Deblurring. We consider non-linear deblur- ring with the same setup as in Chung et al. (2022a). For this task, we compare against DPS (Chung et al., 2022a), RED-Diff (Mardani et al., 2024), MPGD (He et al., 2024),
Models and Datasets: For inverse problems, we conduct experiments on the FFHQ (256 x 256) (Karras et al., 2019)
6

---

Variational Control for Guidance in Diffusion Models
Table 2. NDTM performs on-par/hetter than competing baselines on noisy linear inverse problems. Missing entries indicate unstable performance after multiple tuning attempts Bold: best.
Super-Resolution (4x)
Random Inpainting (90 %)
FFHQ (256 256)
Imagenet (256 x 256)
FFHQ (256 256)
Imagenet (256 x 256)
Method
LPIPS|
FID 4
KID L
LPIPS|
FID
KID 4
LPIPSI
FID 4
KID
LPIPS
FID 4
KID |
DPS
0.061
20.61
0.0029
0.195
30.67
0.0021
0.058
20.24
0.0019
0.152
32.56
0.0023
DDRM
0.116
36.13
0.0183
0.325
52.76
0.0151
0.582
167.57
0.1530
0.791
211.66
0.1517
RED-diff
0.151
41.54
0.0179
0.354
51.83
0.0084
0,430
155,49
0.1370
0.633
218.88
0.1531
MPGD
0.119
28.54
0.0032
0.215
37.39
0.0017
0.658
173.28
0.134
0,908
156.44
0.053
C-IIGDM
0.106
29.61
0.0073
0.270
39.96
0.0024
0,551
137.85
0.1020
RB-Modulation
0.054
18.22
0.0013
0.211
35.26
0.0032
0.091
25.76
0.0026
0.223
46.30
0.0066
NDTM (ours)
0.054
18.99
0.0019
0.158
28.75
0.0011
0.059
20.11
0.0020
0.149
30.43
0.0018
Reference
Degradation
NDTM (Ours)
DMPlug
Table 3. Comparisons on noisy Blind Image Deblurring (BID) for the FFHQ 256x256 256 dataset. NDTM outperforms DMPlug (Wang et al., 2024) by significant margin while requiring order of magnitude less sampling time (reported in minutes/img).
Bold: best. t: N=15, T=200, Ç: N=15, T=400.
Gaussian
Gaussian blur
Motion blur
Method
LPIPS
FIDJ
Time!
LPIPS)
FIDJ
Time!
DMPlug
0.147
69.36
51.24
0.118
72.85
51.13
NDTM (Ours)
0.103
55.15
7.17
0,086
49.99
7.17
NDTM (Ours)
0.083
47.34
18.07
0.063
38.6
18.13
and RB-Modulation (Rout et al., 2025) (which is a special
case of our method with "/- = 1.0, w. = 0, w, = 0). Fig- ure (Right) illustrates the comparison between competing baselines and our proposed method, NDTM, on this task. Qualitatively, we find that DPS is very sensitive to guidance step size and is usually unstable on this task. Moreover, while RED-diff does not have stability issues, it is biased towards generating blurry samples. This is not surprising given their unimodal approximation to the data posterior p(xoly). On the contrary, NDTM generates high-fidelity reconstructions with a stable sampling process. Similarly, our quantitative results in Table 1 validate our qualitative findings as our method outperforms competing baselines perceptual quality for both datasets by a significant margin.
Motion
Figure 2. NDTM outperforms competing baselines on blind im- age deblurring (BID) with Gaussian (top) and Motion (bottom) kernels. NDTM accurately captures most details, while competing methods introduce artifacts in the generated reconstructions.
Figure 2 (Right) illustrates the comparison between DM- Plug and NDTM adapted for this task. Qualitatively, we find that while DMPlug can introduce artifacts in generating reconstructions, NDTM generates high-quality reconstruc- tions. Table 3 further validates our qualitative findings as our method outperforms DMPlug on perceptual quality metrics. More interestingly. while DMPlug is extremely expensive for single image, our method outperforms the former on sample quality by a significant margin while being order of magnitude faster (see Table 3). This illustrates that our sampler has a more efficient way to trade sampling speed for quality. We present additional results in Appendix C.
Blind Image Deblurring (BID). Next, we extend our frame- work to blind image deblurring, maintaining the same setup as DMPlug (Wang et al., 2024). We compare against DM- Plug, which, to the best of our knowledge, is also the state- of-the-art method for this task. Interestingly, adapting our proposed method for blind inverse problems only involves jointly optimizing the unknown blur kernel parameters along with the control u, More specifically, for degradation of the form y = k* * Xo + o with unknown blurring kernel & we update line 11 in Algorithm 1 as Curminal wy uy (x((() k((O) and optimizing for the trainable kernel for each image, as (6+1) t- (p)ate(k( V,c(.))
Linear Inverse Problems. Lastly, we compare competing
7

---

Variational Control for Guidance in Diffusion Models
Reference Style
Prompt: "a cat wearing glasses"
FreeDOM [Yu J
Style: 555.5 CLIP: 32.03
Style: 798.0 CLIP: 27.46
Style: 4:0.2 CLIP: 32,50
Style: 425.7 CLIP: 29.84
Style: 481.7 CLIP: 31.54
Style: 700.5 CLIP:
MPGD [Hc et al.]
Style: 441.7
Style:
Style:
Style:
Style:
Style:
(Ours) NDTM
Style: 389.5 CLIP: 32.18 Style: 668.5
CLIP: 29.26
Style: 359.0 CLIP: 30.40 Style: 398.0 CLIP: 31,12 Style: 357,7 CLIP: 31.68
Style: 562.5 CLIP: 31.32
Figure Style Guidance with Stable Diffusion. NDTM (proposed) provides better tradeoff between Prompt adherence Style adherence using Stable Diffusion 1.4. While baselines can introduce random artifacts in generated samples, NDTM preserves sample quality while exhibiting better style adherence. (Top Panel) Reference Style Images. (Bottom Panel) Samples corresponding to the reference style images in the Top Panel. The CLIP (Radford et al., 2021) score (higher is better) represents the similarity between the generated image and text prompt, indicating prompt adherence. The Style score (lower is better) represents distance between CLIP features for the reference style and the generated image, indicating style adherence. We present additional quantitative results in Table 4,
methods línear inverse problems: (4x) Super-resolution and Random inpainting with 90% masking probability. In addition to the baselines used for the non-linear deblur task,
CL.IP Score f
Style Score
FreeDOM 30.86
508.28
MPGD
30.21
498.85
we also compare against DDRM (Kawar et al., 2022) and C-TGGDM (Pandey et al., 2024b). As illustrated in Table 2, for super-resolution, NDTM outperforms competing base- lines for both datasets. For random inpainting, our method performs comparably with DPS on the FFHQ dataset. How- ever, for more difficult benchmark like ImageNet, NDTM outperforms the next best competing baseline, DPS, on this task. We present additional qualitative results for linear inverse problems in Appendix C
(Ours) NDTM
31.34
475.62
Table 4. Quantitative comparison hetween NDTM and other base- lines on style guidance generation using Stable Diffusion 1.4. NDTM exhibits better prompt (see CLIP score) and style adher- ence (see Style Score) over competing baselines.
image. We follow the same setup MPGD and pretrained Stable Diffusion 1.4 text-to-image model. We specify the terminal cost for NDTM in Eq. (22) and com- pare with MPGD (He et al., 2024) and FreeDOM (Yu et al., 2023) on this task. Figure and Table 4 demonstrate that NDTM exhibits better balance between prompt and style
5.2. Style Guidance
The goal in style guidance is to generate samples that ad- here to a specified text prompt and the style reference
8

---

Variational Control for Guidance in Diffusion Models
LPBS
LIMPS
PSNR
LPPS
160-
Rartina
140
PSNit (400)
L20
1.P1PP
1271P%
Buntiin
PRNNL
PSNK
40-
20-
(a) Terminal Weight (wr)
(b) Guidance Weight (7)
(c) Optimization Steps (N)
(d) Runtime N
Figure 4. Impact of different design choices in NDTM on Distortion (PSNR) and Perception (LPIPS) for the non-linear deblur task. (a.
b) The extent of guidance can be jointly controlled by varying the terminal loss weight (wr) and the weight (7). (c. Compute vs quality can he traded off by jointly varying the numher of optimization steps (N) and the number of diffusion steps.
adherence over competing baselines.
Degraded
5.3. Ablation Studies
Next, we analyze the impact of different design choices in NDTM on the perception (LPIPS) and distortion (PSNR) quality for the non-linear deblur task on ImageNet.
Impact of Guidance. Since the terminal cost weight wr and the parameter y affect the optimization of the varia- tional control parameters u, we analyze their impact on sample quality. From Fig. 4a, we observe that increasing the terminal weight wT leads to improvement in both perceptual and distortion quality. However, in the limit of WT + oc (i.e., where the regularization terms in Eq. 17 can be ignored), the perceptual quality degrades, which high- lights the importance of the transient cost in our framework. Similarly, increasing y also leads to an improved sample quality. However, large 7 can also lead to overshooting.
Figure 5. The optimal variational controls hierarchically refine image features over time. (Top Row) Non-Linear Deblur (Bottom Row) Random Inpainting. (Left to Right) We visualize optimal controls at different times to > t > t2 >> ts in diffusion sam- pling, progressively capturing coarse to fine details.
6. Conclusion
Our proposed framework offers a principled way to guide a pretrained diffusion model while respecting an external cost through the lens of variational optimal control. Our empiri- cal results suggest that optimizing each diffusion step allows for more flexibility in guidance compared to commonly used approximations of the diffusion posterior.
Impact of Optimization Steps. It is common to trade sam- ple quality for the number of sampling steps in diffusion models. Interestingly, NDTM provides a complementary axis to achieve this tradcoff in the form of adjusting the number of optimization steps per diffusion update. We illus- trate this in Fig. 4(b), where for fixed sampling budget of 50 diffusion steps, NDTM can achieve better reconstruction quality by increasing the number of optimization steps (N). However, since the runtime increases linearly as N grows (see Figure 4(d)), a practical choice depends on the avail- able compute. We find that for this task, N=2 provides a favorable tradeoff between sampling time and quality and, therefore, use it for state-of-the the-ar--tttttttt art comparisons on the Ima- geNet dataset in Table 1.
While our method adapts well to diverse tasks, there remain several interesting directions for future work. First, more thorough theoretical investigation into the optimization dy- namics of the proposed method and integration with existing methods for fast diffusion sampling (Pandey et al., 2024a) could help alleviate the sampling costs of our method. Sec- ond, our method is only one instantiation of our framework, leaving ample room for exploration in investigating novel variational parameterizations and refining cost functions which could further enhance the flexibility of our approach.
Control Visualizations We visualize the optimal controls In in Figure 5. We observe a hierarchical refinement of image features over time. More specifically, the control inference captures global structure at the start of diffusion sampling and gradually refines local details (like edges), thereby encoding high-frequency information later steps.
Lastly, in broader sense, work gives another example of the intricate connections between test-time adaptation of diffusion models and Bayesian inference; this view may enable future generative models to perform more complex inference tasks such as hierarchical modeling, and quantify- ing their uncertainty (Jazbec al., 2025).

---

Variational Control for Guidance in Diffusion Models
Acknowledgments
ISSN 2835-8856. net/ pett/forumiid=oY11 pee:/Foormmm?.doo..333ppp...
URL https:/ oppnrevieneeee.eeee
We thank Justus Will for additional discussions and feed- back. This project was funded through support from the Chan Zuckerberg Initiative. Additionally, Stephan Mandt acknowledges funding from the National Science Founda- tion (NSF) through NSF CAREER Award IIS-2047418, IIS-2007719, the NSF LEAP Center, and the Hasso Plat- tner Research Center at UCL. Parts of this research were supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Busi- ness Center (DOI/IBC) contract number 140D0423C0075. The U.S. Government is authorized to reproduce and dis- tribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.
Binkkwwki,, M., Sutherland, D. Arbel, M., and Gretton, A. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL. https:// openreview wweee/oorumiddr110000
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: A review for statisticians. Journal of the Amer-
ican Statistical Association, 1 112(518):859
April
2017. ISSN 1537-274X. doi: 10. 10.1080001621459.2011 1285773. URL http: dx doi wwg.oo.gg//.oooo..ooooooo 1080/ 01621459. 2017. 011857772.
Boys, B., Girolami, M., Pidstrigach, J., Reich, S., Mosca, A.. and Akyildiz, O. D. Tweedie moment projected diffusions for inverse problems. Transactions Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: openreview net forum? fettt/formmiddduuuu 4imndiddi4niqrT Fea- tured Certification.
Impact Statement
Despite good sample quality in limited sampling budget, restoration can sometimes lead to artifacts in the generated sample, which can be undesirable some domains like medical image analysis. Moreover, since our method relies on defining terminal cost, can be used for the synthesis of harmful content using ill-defined cost objective.
Chen, T., Gu, J., Dinh, I.,, Theodorou, E., Susskind, J. M., and Zhai, S. Generative modeling with phase stochastic bridge. In The Twelfih International Confer- ence on Learning Representations, 2024. URL https: / //openreview nwtp//neeeeeeeeeeeeee forummpidttttcjjooddee
Chung, H., Kim, J., Mecann, M. T., Klasky, M. L., and Ye, J. C. Diffusson posterior sampling for general noisy inverse problems. In The Eleventh International Confer- ence on Learning Representations, 2022a.
References
Albergo, M. and Vanden- E. Building normaliz- ing flows with stochastic interpolants. In ICLR 2023 Conference, 2023.
Chung, H., Sim, B., and Ye, J. C. Come-closerddifusee faster: Accelerating conditional diffusion models for in- verse problems through stochastic contraction. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 12413-12422, 2022b.
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.0873 2023.
Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applica- tions, 12(3):313-33 1982. ISSN 0304-4149.
Daras, G., Chung, H., Lai, C.-H., Mitsufuji, Y., Ye, J. C., Milanfar, P., Dimakis, A. G., and Delbracio, M. survey diffusion models for inverse problems, 2024. URL https: wwaxiiivoorggo ats///bs/2211// 00083.
doi: http://0o.or////0.0o000044444440 com/ URL
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference computer vision and pattern recognition, pp. 248-255, Ieee, 2009.
Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, R., Goldblum, M., Geiping, J., and Goldstein, T. Uni- versal guidance for diffusion models. In The Twelfih International Conference on Learning Representations, 2024. URL https::tp...............t http///pnrreeieeeeeeeeeeee net/ forum? id Id-pzpWBbnwiJ.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Berner, J., Richter, L., and Ullrich, K. An optimal con- trol perspective on ceftusion-baseer generative model- ing. Transactions on Machine Learning Research, 2024.
Efron, B. Tweedie's 's formula and selection bias. Journal of the American Statistical Association, 106:1602-1614 12 2011. doi: 10.1 198/jasa.200 2011.tmm 1.tm11181 1181.
10

---

Variational Control for Guidance in Diffusion Models
He, Y., Murata, N., Lai, C.-H., Takida, Y., Uesaka, T., Kim, D., Liao, W.-H., Mitsufuji, Y., Kolter, J. Z Salakhutdinov, R., and Ermon, S. Manifold preserving guided diffusion. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.oo net forum? o
2005/11/p11011, URL http:// doi 10. 1058/
Kappen, H. J., Gómez, V., and Opper, M. Optimal control as graphical model inference problem. Machine Learning, 87(2):159-182, February 2012. ISSN 1573-0565, doi: 10. 10.10077/110994--12-577777 URL bttp:// //xx/xxooo/ .oii. org/10. 101000000//1// 012-5111-222221
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401 1001-4410, 2019.
Ho, J. and Salimans, T. Classifier-frre diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion diffusion-based generative models. Advances in Neural Information Processing Systems, 35: 26565-26577 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840- 118840-6551,,.1 2020.
Kawar, B., Elad, M., Ermon, S., and Song, J. Denoising diffusion restoration models. Advances in Neural Infor- mation Processing Systems, 35:23593-2336 2022.
Holderrieth, P., Havasi, M., Yim, J., Shaul, N., Gat, I., Jaakkola, T., Karrer, B., Chen, R. T. Q, and Lip- man, Y. Generator matching: Generative modeling with arbitrary markov processes. In The Thirteenth International Conference on Learning Representations, 2025. URL httpshttt///// Ad=RuP17cJt22.
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL 1412.69 6980.
abs/
Levine, S. Reinforcement learning and control as prob- abilistic inference: Tutorial and review, 2018. URL https: http///axivv.//// abs 110510000990
Huang. Y. Ghatare, A., Liu, Y., Hu, Z.,, Zhang, Q., Sastry, C. Gururani, S., Oore, S., and Yue, Symbolic music generation with non-differentiable rule guided diffusion. In ICML, 2024. URL httpp::/tpppeeeeeeeeee / forum? hiw.mm:idgg@g_aAA-K0.00
Li, A., Boyd, A., Smyth, P., and Mandt, S. Detecting and adapting to irregular distribution shifts in bayesian online learning. Advances in neural information processing systems, 34:6816-6822 2021.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M.. and Le, M. Flow matching for generative modeling. In International Conference Learning Representations, 2023. URL https: /openreview n id=PquMRDCJTSt.
Janati, Y., MOUFAD, B., Durmus, A. O., Moulines, E., and Olsson, J. Divide-and con-corueer posterior sampling for denoising diffusion priors. In The Thihth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:ttt... htttpp:://ppeeeeee http///ppeeeeeeeeee/e//e/ee hwwp:eeetttooeeeeeee net/forum? forum? id=BOrut 7M2X7.
Liu, X., Gong, C., and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. The Eleventh International Conference on Learning Representations, 2023. URL https: opw.revee..eeeeeeeeeeee.. net fet:/formmiic http:/orruumiiddxjjjiiiiv wwwnwTw5T.....
Jazbec, M., Wong-Toi, E., Xia, G., Zhang, D., Nalisnick, E., and Mandt, S. Generative uncertainty in diffusion models. In Conference on Uncertainty in Artificial In- telligence (UAI), 2025. URL https::///aaatt orgg abs/2502.22 20946.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 11461-11471 June 2022.
Kappen, H. Stochastic optimal control theory. ICML., Helsinki, Radbound University, Nijmegen, Netherlands, 2008.
Mardani, M., Song, J., Kautz, J., and Vahdat, A. A vari- ational perspective on solving inverse problems with diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //opprreviee eww net forum? d=1 Y04EE3SPB.
Kappen, H. J. Path integrals and symmetry breaking for opti- mal control theory. Journal of Statistical Mechanics: The- Novem- ber 2005. ISSN 1742-5468. doi: 10. 10.1088/1742-5466
ory and Experiment, 2005 11::111111PP110011111 2000(11):P11011

---

Variational Control for Guidance in Diffusion Models
Nachmani, E., Roman, R. S., and Wolf, L.. Denoising dif- fusion gamma models, 2021. URL httttt:s///axa. /agg/bbs/en 2110. 05948.
The Thirteenth International Conference Learning Representations, 2025. URL https: /ppprevven...eeeeeeeeeee net / fow.mmidd-nIINNGSA...
Pandey, K. and Mandt, S. A Complete Recipe for Diffusion Generative Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2238-4249,, Los Alamitos, CA. USA, October 2023, IEEE Computer Society. doi: 10.1 10.11099CCCV55100222223.. 003933. URL /ttp://doiii/.. attaps//ool..ieeccooptttrssssss 10. 19..1199/20 http0//0CCCV1000000000000 20000/..22...0
Saharia, C.,, Chan, W., Chang. H., Lee, C., Ho, J., Salimans,
T., Fleet, D., and Norouzi, M. Palette: Image-to-imagee diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Saleh, and Flgammal, Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv: 00855, 2015.
org/
Pandey, K., Rudolph, M., and Mandt, S. Efficient inte- grators for diffusion generative models. In The Twelfth International Conference on Learning Representations, net/forum?
Dicksiistt,,,, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamics. International Conference on Machine Learning, pp. 2256-2265, PMLR, 2015.
2024a. URL htttps:///ppeeeeetttttt id=oA4 foXO5GF.
Pandey, K., Yang, R., and Mandt, S. Fast samplers for inverse problems in iterative refinement models. In The Tiihtt- Annual Conference on Neural Infor- mation Processing Systems, 2024b. URL https:// openreview net forum? dddiisAIvttaaa
Song, J., Meng, C., and Ermon, S. Denoising diffu- sion implicit models. In International Conference on Learning Representations, 2021a. URL. https:// openreview net Seerumidd-Stiiiiie
Song, J., Vahdat, A., Mardani, M., and Kautz, J. Pseudoinverse guided diffusion models for inverse prob- lems. In International Conference on Learning Represen- tations, 2023. URL https: / openreview net/ forum? id= 9eatimiddddd__a_______ 9_____MMMMMKK---
Pandey, K., Pathak, J., Xu, Y., Mandt, S., Pritchard, M., Vah- dat, A., and Mardani, M. Heavy-tailed diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /cppneevie..... net/ forum? widoozoooooeeee
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: // /ooeereview net forum? ? d=di 522R8xgf.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Pokle, A., Muckley, M. J., Chen, R. T. Q., and Karrer, B. Training-free linear image inverses via flows. Transac- tions on Machine Learning Research, 2024. ISSN 2835- http:/iwww.eeee..reeeeeeeee
Song. Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.). Advances Neural Information Processing Systems, 2021b. URL htts::::: wppp//veerreeeeeeeeeeeeeee .nti//.we/////eeee.em.. forum?
8856. URL https: Ad-PLIt3a4yI
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763, PmLR, 2021.
Tran, P., Tran, A. T., Phung, Q., and Hoai, M. Explore image deblurring via encoded blur kernel space. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11956-11965, June 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffiusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Vincent, P. connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661 1674, 2011.
Rout, L., Chen, Y., Ruiz, N., Kumar, A., Caramanis, C., Shakkottai, S., and Chu, W.SS RB- RB-modulation: Training- free stylization using befdee modulation. In
Wang, H., Zhang, X., Li, T. Wan, Y., Chen, T., and Sun, J. DMPlug: A plug-in method for solving inverse prob- lems with diffusion models, In The Thirty-eighth Annual
12

---

Variational Control for Guidance in Diffusion Models
Conference on Neural Information Processing Systems, 2024. URL btttts:://ppereeeeee net. forum? id-81IFFsfQUj.
Wang, Y., Yu, J., and Zhang, J. Zero-shot image restora- tion using denoising diffusion null-space model. In The Eleventh International Conference Learning Represen- tations, 2023. URL https: //t:///pppee /pw.reveeee.....o n forum? weiiddiiiieeMMMTT
Ye, H., Lin, H., Han, J., Xu, M., Liu, S., Liang, Y., Ma, J., Zou, J., and Ermon, S. TFG: Unified training-free guid- ance for diffuuson models. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. URL https:://pperreiieeeeee/ net forum? id-N8YbGX98ve.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-image gen- eration. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Yu, J., Wang, Y., Zhao, C., Ghanem, B., and Zhang, Freedom: Training- fraee grrrgg-ggddddddd conditional diffu- sion model. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 23174 23184, October 2023.
Zhang, C., Bütepage, J., Kjellström, H., and Mandr, S. Ad- vances in variational inference. IEEE transactions on pattern analysis and machine intelligence, 2000:200
2026, 2018a.
Zhang. R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586586555 586-595, 2018b.
Zhou, M., Chen, T., Wang, Z., and Zheng, H. Beta dif- fusion. In Thirty-seventh Conference on Neural In- formation Processing Systems, 2023. URL https: //openreview.o wttp:eiwwwwwweeeeeeeeeeeeeeeeeee f (tttorrmmmiiiSiooooe mm4m1..
13

---

Variational Control for Guidance in Diffusion Models
A. Proofs
A.I. Simplification of the NDTM Objective for DDIM
We restate the theoretical result for convenience.
Proposition A.l. For the diffusion posterior parameterization in Song et al. (2021a), the objective in Eq. 16 can be simplified as (see proof in Appendix A.I),
S S ull2 ( - cx( 1) + (T(((
(23)
Wya and v1 - of V Val-1 (1 oa) Vai
where x, Xx + 7,0, is the guided state and the coefficients Kit time-dependent scalars.
06-1
Van
Proof. In the case of DDIM (Song al., 2021a), the diffusion posterior is parameterized as (Eqn. 12 in Song al. (2021a)).
(x,,)) vat-1
Xx + 06-1 of v 0 - at) Voi
6x ().
(24)
Vax
where the diffusion noising process is parameterized p(x: (o) N (Va:xo, (1 at) La) and (x(, t) is pretrained denoiser which models E[[[x,] and intuitively predicts the amount of noise added to Xg for a given noisy state x, at time t. Additionally, for notational convenience, we denote the coefficient of the denoiser in Eq. 24 as тy. Following Song et al. (2021a), the coefficient a is further defined as,
at
0t
a(-1) (1 (t.)
(1
(25)
Org-1
It follows that,
Xt + (xco(x.t)) t)
(26)
Va
no(x; + (),,)) = yau-1
(xt + ?ue) + Treo((x + 7u,,
(27)
Va:
va-1
at-1
X
+
a u, + T:00(x t,,,)
(28)
Vas
where we denote the coefficient of the control signal u, the above equation as K, for notational convenience. Consequently, the NDTM cost in Eq. 16 can be simplified for the DDIM posterior parameterization Eq. 24 as,
Ho (x, + yu, t) na (x,, 1) wT w- (X+h]
(29)
||| ka + Tr [x+ + yu, t) - co (x,, 1)) _ ur + (x{)]
(30)
50 | 7" (xx + tu, c (x,, 1) wy (x
(31)
where (i) follows from the triangle inequality. This completes the proof.
A.2. Continuous-Time Diffusion Trajectory Matching
Analogous to the discrete case, we represent unguided diffusion dynamics using the following continuous-time reverse diffusion dynamics (Anderson, 1982; Song et al., 2020),
dx, [[()xx g(1)2 80) so(xxxxe)) 1) + g())dw.,
(32)
14

---

Variational Control for Guidance in Diffusion Models
where sø (x;;; 1) is a pretrained score network. Similarly, parameterize the guided continuous dynamics by inserting the
control non-linearly into the score function follows,
dx. = f(t)x; -gt)) vot))ss(xx, t) dt + g(t)dwr.
(33)
Denote the unguided path measure as u (x - 0)) and the guided path measure as μx((T - +0) u(T - +)).
Then, the optimal control problem reads, in analogy to Eq. (8):
C(XT, u(T + 0)) WT
By
(xx))
(KI(u(x(T - u
u(T - -)) ( -0) 0)
(34)
Terminal Cost Cse
Transient Cr
By (Song et al., 2021b, Theorem in Appendix A) (which follows from an application of Girsanov's Theorem), the transient cost reads:
Cu DKL (u(I(T 0)xx, u(T - 0)) 1xx(T( 0](x))
(35)
1 2 / ((1) Bx,) ue, () so 11,
(36)
Taking the approximation that the control signal is optimized greedily, find Eq. (12).
A.3. Extension Flow Matching Models
For continuous flow matching models (Lipman al., 2023; Albergo Vanden-Eijnneen, 2023; Liu al., 2023) with vector field vo(x,, (),
dx: (x(x,, t)dt,
(37)
we insert the control signal into the dynamics through an additional dependence of the velocity field:
dx,
u t).
(38)
dt
Since flow matching uses the squared loss, it is natural to regularize deviation from the unguided trajectory in terms of the velocity field:
Cr = / |100(xe ur, (x(,,, 2
(39)
B. Implementation Details
In this section, we include practical implementation details for the results presented in Section 5.
B.1. Task Details
B.1.1. INVERSE PROBLEMS
Here, we describe the task setup in more detail.
Superresolution (x4): We follow the setup from DPS (Chung et al., 2022a), More specifically,
~ L/y
(40)
where S represents the bicubic downsampling matrix with downsampling factor f. In this work, we fix f to 4 for both datasets.
Random Inpainting (90%) We use random inpainting with a dropout probability of 0.9 (or 90%). For this task, the forward model can be specified as,
(41)
where M E {0, dxd the masking matrix.
15

---

Variational Control for Guidance in Diffusion Models
Non-Linear Deblurring We the non-linear deblurring setup from DPS. More specifically, the forward operator Fø (modeled using neural network) for the non-linear deblurring operation. Given pairs of blurred and sharp images, {ay, y one can train a forward model estimator (Tran et al., 2021),
arg min y Fa(m;, Y:)11
(42)
where G extracts the kernel information from the training pairs. At inference, the operator 9 can instead be replaced by Gaussian random vector g. In this case, the inverse problem reduces to recovering æ; from y,- In this work, we directly adopt the default settings from DPS.
Blind Image Deblurring (BID) We directly adopt the setup for blind image deblurring from DMPlug (see Appendix C.4 in Wang et al. (2024) for more details). More specifically, in the BID task, the goal is to recover the kernel & in addition to the original signal Xg such that,
y lo 37 + 0y²
(43)
In this work, we adapt the default settings from DMPlug. For BID (Gaussian), the kernel size is 64 64 with the standard deviation set to 3.0. For BID (Motion), the kernel intensity is adjusted to 0.5.
B.1.2. STYLE GUIDANCE WITH STABLE DIFFUSION
In the context of text-to-image models like Stable Diffusion, the goal of style guidance is to generate a sample that simultaneously adheres well to given text prompt and style features from a reference image. More specifically, given a reference style image T, a pretrained feature extractor (like CLIP) F, a pretrained decoder D such that xj = D(2)). we define & (xt) as,
e(x5)= = [[9(F(()) ((F
(44)
where 9 denotes the Gram-matrix operation and ||-|| F denotes the Frobenius norm. In this formulation, the pretrained text-to-image diffusion model works as a generative prior, p(x[t), where is text prompt embedding, and the goal is to generate samples from the posterior p(x[r, t)
B.2. Inverse Problems- Task Specific pyperaarameters
Here, we provide a detailed overview of different hyperparameters for the baselines considered in this work for the inverse problem tasks. We optimize all baselines and our method for the best sample perceptual quality. We use the official code implementation for RED-Diff (Mardani et al. 2024) at https: http:::://ttttubttttt wwwp////w..coocco/o///ooooaaaaae /RED- RED-diff. ht https: / t/itbbb.. thub. www/maoot- latp/tttllbpppppppa c- gdm, and https: /tttt/b...ttttggg//g ctw/ wutmm//uuuummm//ml/u for running all competing baselines.
B.2.1. DPS (CHUNG ET AL., 2022A)
We adopt the DPS parameters from Mardani et al. (2024). More specifically, we fix the number of diffusion steps to 1000 using the DDIM sampler. We set 17 0.5 for all tasks. Following Chung et al. (2022a), we set,
(45)
1y A
Table 5 illustrates different hyperparameters for DPS on all tasks for the FFHQ and ImageNet datasets.
B.2.2. DDRM (KAWAR ET AL., 2022)
Following Kawar al. (2022), we fix 1) = 0.85, 1p = 1.0, and use the DDIM sampler with the number of diffusion steps set 20 across all linear inverse problems.
B.2.3. C-ITGDM (PANDEY ET AL., 2024B)
We set the number of diffusion steps to 20 for all tasks. It is also common to contract the reverse diffusion sampling for better sample quality by initializing the noisy state as proposed in Chung et al. (2022b). We denote the start time as т. We re-tune C-IGDMM for the best perceptual quality for all linear inverse problems. Table 6 illustrates different hyperparameters for linear inverse problems. We find that C-IIGDM fails to recover plausible images for the random inpainting task after numerous tuning attempts.
16

---

Variational Control for Guidance in Diffusion Models
Table 6. C-TGGDM hyperparameters used for different tasks. We find that C-LIGDM fäils to recover plausible images for the random inpaint- ing task after numerous tuning attempts.
Table 5. DPS hyperparameters for different tasks
FFHQ
ImageNet
Task
FFHQ
ImageNet
Super- Super-Resolution (4x)
1.0
1.0
Task
X
W
7
0
W
T
Random Inpainting (90%)
1.0
1.0
Super-Resolution Resolution (4x)
-0.4
4.0
0.4
-0.4
4.0
0.4
Mon-Linear Deblur
0.3
1.0
Random Inpainting (90%)
Table 7. RED-Diff RED-Diff hyperparameters used for different tasks.
FFHQ
ImageNet
Task
lr
X
Ir
X
Super-Resolution (4x)
0.5
1.0
0.5
0.4
Random Inpainting (90%)
0.5
0.25
0.5
0.25
Non-Linear Deblur
0.5
0.25
0.5
0.25
B.2.4. RED-DIE (MARDANI ET AL., 2024)
We set O with linear weighting schedule and Ir 0.5, À = 0.25, and perform 50 diffusion steps using the DDIM sampler for all tasks across the FFHQ and ImageNet datasets. We highlight different hyperparameters in Table 7.
B.2.5. MPGD (HE ET AL.,, 2024)
Following the optimal settings in He et (2024), we use the setting MPGD w/o proj using the DDIM sampler with 100 diffusion steps and guidance scale set to 5.0 for all tasks.
B.2.6. NDTM (OURS)
We use the Adam optimizer (Kingma & Ba, 2017) with default hyperparameters, fixing the learning rate to 0.01 for updating the control u, across all tasks and fixing the kernel learning rate in the BID task to 0.01. We refer to the loss weighting scheme in Eq. 16 as "DDIM weighting". Moreover, we use linear decay for the learning rate. We perform 50 diffusion steps using the DDIM sampler across all datasets and tasks. We tune the guidance weight 7, the number of optimization steps N, loss weighting (wr, w,, we), DDIM y and the truncation time (Chung et al., 2022b) for best performance across different tasks. All these hyperparameters are listed in Table 9.
B.2.7. 1B-MODULATION (ROUT ET AL., 2025)
Since RB-Modulation is special case of NDTM with 7 = 1.0 and w,, w. = 0. we re-run NDTM for different tasks with these settings, keeping all other hyperparameters fixed to report results for RB- RB-MMoulatioo.
B.3. Style Guidance Hyperparameters
B.3.1. MPGD (HE ET AL., 2024)
Following He et al. (2024), use the DDIM sampler with 100 steps and 17 1.0 without the time reversal (Lugmayr et al., 2022) trick for fair comparisons. We set 17.5 and the classifier-free guidance scale to 7.5
B.3.2. FREEDOM (YU ET AL., 2023)
Following Yu et al. (2023), we use the DDIM sampler with 100 steps and r7 = 1.0 without the time reversal trick. We use the classifier-free guidance scale of 5.0
17

---

Variational Control for Guidance in Diffusion Models
Table BID hyperparamettes. for NDTM.
FFHQ
Task
7
WT
0%
Me
BID (Gaussian)
15
1.0
0.7
1000
50
ddim
ddim
BID (Motion)
15
1.0
0.7
1000
50
ddim
ddim
Table 9. NDTM hyperparameters for different tasks.
FFHQ
ImageNet
Task
N
7
1
WT
We
N
Y
1J
WT
W
Super-Resolutton (4x)
5
1.0
0.7
400
50
ddim
ddim
2
2.0
0.1
600
50
ddim
ddim
Random Inpainting (90%)
2
4.0
0.2
500
1
0
0
4.0
0.0
600
50
ddim ddim
Non-Linear Deblur
5
5.0
0.1
400
0
0.1
600
50
ddim ddim
B.3.3. NDTM
For NDTM, we use the DDIM sampler with 50 steps and 1.0. We set the control learning rate to 0.002 with the Adam
optimizer. The loss weightings are set to wrr = 1.0, wt = 0, t0s = 0 with 4.0, N and a classifier-free guidance scale of 5.0
C. A Addiiinna Experimental Results
C.1. Evaluation on Distortion Metrics
In this work, primarily optimize all competing methods for perceptual quality. However, for completeness, compare the performance of our proposed method with other baselines on recovery metrics like PSNR and SSIM. Tables 12 and 13 compare our proposed method, NDTM, with competing baselines for linear and non-linear inverse problems. We find that NDTM performs on par with other methods for the super-resolution task. However, for random inpainting and non-linear deblur, NDTM outperforms competing methods terms of distortion metrics like PSNR. Since NDTM also outperforms existing baselines in terms of perceptual quality (see Table 2), our method provides a better perrepptonn tradeoff.
C.2. Runtime
Below, we compare diffcrent methods in terms of the wall-clock time required for running on a single image for the superresolution task. From Table 10, we observe that while our method requires an inner optimization loop, it is still faster than common baselines like DPS and DMPlug (see Table 11).
Table 10, Runtime comparisons for different baselines vs NDTM for super-resolution task both datasets. The runtime numbers are in wall-clock time (scconds) and tested on single RTX A6000 GPU.
FFHQ (256 x 256)
Imagenet (256 x 256)
DPS
RED-diff C-HIGDM DDRM NDTM (Ours)
DPS RED-diff C-IIGDM DDRM NDTM (Ours)
Runtime (secs/ Img) 199.1
5,8
3.68
1.3
13.6
399.3
7.1
16.4
2.4
38.3
18

---

Variational Control for Guidance in Diffusion Models
Table Runtime comparisons for DMPlug baseline vs NDTM for blind image deblurring (BID) task on FFHQ dataset. The runtime
numbers are in wall-clock time (minutes) per image and tested on single RTX A6000 GPU.
Method
Gaussian blur
Motion blur
Timel
Timel
DMPlug
51.24
51.13
NDTM (Ours)
7.17
7.17
NDTM (Ours)
18.07
18.13
Table 12. Comparison between NDTM and existing methods for Linear IPs on distortion metrics like PSNR and SSIM. Missing entries indicate that the method unstable for that specific task. Bold: best.
Suser-Resolution (4x)
Random Inpainting (90 %)
FFH (256 x 256)
Imagenet (256 x 256)
FFHQ (256 256)
Imagenet (256 256)
Method
PSNRT
SSIM
PSNRT
SSIMT
PSNRT
SSIM
PSNRT
SSIM1
DPS
29.06
0.832
23.61
0.676
27.76
0.832
20.96
0.657
DDRM
30.12
0.864
24.15
0.701
17.34
0.371
15,91
0.257
RED-diff
27.67
0.720
24.06
0.685
20.84
0.581
18.63
0.466
C-IGGD
27.93
0.773
23.20
0.631
MPGD
26.07
0.715
21.83
0.587
11.34
0.076
10.26
0.025
RB-M Modulation
29.12
0.831
23.41
0.674
26.90
0.810
21.31
0.632
NDTM (ours)
29.06
0.833
23.12
0.674
28.03
0.834
21.34
0.665
Table 13. NDTM outperforms existing methods for Non-linear deblur on distortion metrics like PSNR and SSIM. Bold: best.
FFHQ (256 x 256)
ImageNet (256 256)
Method
PSNRT
SSIM1
PSNRT
SSIM
DPS
8.12
0.262
6.67
0.156
RED-diff
24.88
0.717
21.88
0.623
MPGD
18.24
0.406
17.02
0.261
RB-Modulation
29.39
0.846
22.11
0.612
NDTM (ours)
30.64
0.874
24.41
0.732
19

---

Variational Control for Guidance in Diffusion Models
Original
Degraded
NDTM (Ours)
RED-DifT
RB-Modulation
Figure Qualitative comparison between NDTM top competing baselines (see Table on the Non-Linear Deblurring task for ImageNet. NDTM better recovers the structure of the image compared to other baselines. Best viewed when zoomed in.
20

---

Variational Control for Guidance in Diffusion Models
Original
Degradation
NDTM (Ours)
MPGD
Red-Diff
RB-Modulation
SOUREY
Figure 7. Qualitative comparison between NDTM and top competing baselines (see Table 1) on the Non-Linear Deblurring task FFHQ. NDTM better recovers the structure of the image compared to other baselines. Best viewed when zoomed in.
21

---

Variational Control for Guidance in Diffusion Models
NDTM (Ours) (M-15, T-400)
NDTM (Ours)
Original
Degraded
(M-15, T-200)
DMPlug
L
Figure Qualitative comparison between NDTM and competing baseline (DMPlug) on the blind image deblurring task (see Table 3). NDTM better recovers the details and structure of the image compared to the baseline. We find DMPlug introduces noisy artifacts and blurry images in some samples. Best viewed when zoomed in.
22

---

Variational Control for Guidance in Diffusion Models
Original
Degraded
NDTM (Ours)
DPS
RED-Diff
RB-Modulation
Figure 9, Qualitative comparison between NDTM and top competing baselines (See Table 2) on the Random Inpainting (90%) Task for ImageNet. NDTM performs better or/par compared to other baselines. Best viewed when zoomed in.
23

---

Variational Control for Guidance in Diffusion Models
Original
Degradation
NDTM (Ours)
Red-Diff
RB-Modulation
DPS
MAK
MAN
MAR
AA
MA
Figure Qualitative comparison between NDTM and top competing baselines (See Table 2) on the Random Inpainting (90%) Task for FFHQ. NDTM performs hetter or/par compared to other baselines. Best viewed when zoomed in.
24

---

Variational Control for Guidance in Diffusion Models
Original
Degraded
NDTM (Ours)
DPS
C-IIGDM
RB-Modulation
Figure 11. Qualitative comparison between NDTM and top competing baselines (See Table 2) on 4x super-resolution task for ImageNet. NDTM performs better or/par compared to other baselines. Best viewed when zoomed in.
25

---

Variational Control for Guidance in Diffusion Models
Original
Degradation
NDTM (Ours)
RB-Modulation
MPGD
C-IIGDM
DPS
Figure Qualitative comparison between NDTM top competing baselines (See Table on super-resolution task for FFHQ. NDTM performs better or/par compared to other basclines. Best viewed when zoomed in.
26


Reviews:
[{"review_id": "AkgczZ3tNK", "reviewer": "Reviewer_DxJn", "summary": "The paper proposes to optimize the guidance signal with three losses $C_{\\text{score}}, C_{\\text{control}} and C_{\\text{terminal}$. After that, the guidance signal is utilized in the sampling process as normal guidance. The guidance signal is updated through the greedy scheme.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address specific concerns as follows in the order of sections:\n\n> The method might be too expensive. The running time cost should be provided\n\n> Please compare the running time of the algorithm with other CFG methods.\n\nRuntime is often faster than the closest competitor. See the runtimes in Table 2 in the main text and Table 9 in Appendix C.2, where we list runtimes once per hyperparameter configuration. In detail, our method is about one order of magnitude faster than DPS, the closest competitor for superresolution and random inpainting. While our method is indeed slower than for example Red-diff or DDRM, it always improves the sample quality. Note that we did optimize all hyperparameters for sample quality for all methods. We agree that reducing the sampling time is a crucial improvement for guiding diffusion sampling, but this limitation applies to all high-quality competitors.\n\n> What is the value of N in Algorithm 1? \n\nWe provide complete details of hyperparameters for our method and baselines in Tables 4,5 and 6 in the Appendix. To summarize, we set N=50 for linear and non-linear inverse problems and N=200 for blind inverse problems.\n\n> The evaluation does not have generation task for example label-condition generation, text2image generation. This hinders the evaluation of the work. [...]  Please include these tasks in the paper.\n\nOur framework indeed generalizes to non-linear tasks such as those pointed out by the reviewer. We demonstrate this by simply adapting the form of the terminal cost for Style Guidance generation and text-conditioned generation.\n\n**Style Guidance Generation**: Following the experimental setup in MPGD [He et al.], we evaluate our method on Style Guidance using Stable Diffusion 1.4. Please refer to our qualitative results at this URL: **[https://imgur.com/a/wEDqoaR](https://imgur.com/a/wEDqoaR)**. We further quantitatively evaluate our method on this task on 1000 (reference image, prompt) pairs and compare with FreeDOM [Yu et al.] and MPGD in the accompanying [**Table 1**](https://imgur.com/a/jTn0hQ2). Our quantitative and qualitative results suggest that our NDTM can be applied to this task and outperforms competing baselines on prompt (CLIP score) and style adherence (Style Score) while exhibiting better visual perceptual quality.\n\n**Text Guidance with CLIP**: Following the experimental setup in MPGD [1], we further present qualitative results on text-conditional generation using a pretrained diffusion model on the CelebA-HQ dataset, which can be accessed using this link: **[https://imgur.com/a/5jlnDLo](https://imgur.com/a/5jlnDLo)**. Similar to StyleGuidance, NDTM can be used to generate samples that adhere well to not only simple prompts (like Blond hair) but also more complex prompts such as “a person with blonde hair and a goatee”.\n\nTherefore, we emphasize that our method can be applied to more general guidance scenarios. For brevity, we will include additional qualitative results for these tasks with their experimental settings (like the form of the terminal cost) in a revised version of our paper. Note that we selected the best hyperparameters for the selected baselines, just like for all other experiments.\n\nWe also urge the reviewer to check out our response Reviewer UY5V (see point Re. Limited Empirical Comparisons) which presents additional empirical comparisons with additional baselines on 4x super-resolution and non-linear deblur tasks\n\n> There are recent advances in classifier-free guidance but there is no discussion or comparison in the main paper. [see also] Essential References Not Discussed\n\nWe thank the reviewer for the interesting pointers, which we will include in a dedicated section on classifier-free guidance in the related work section. Note that the two approaches are complementary: Classifier-free guidance is based on retraining a diffusion model using example training data, training-free guidance is based on a pretrained model and a cost function. We think an extensive evaluation of different paradigms is beyond the scope of this work.", "claims_and_evidence": "no problem", "methods_and_evaluation_criteria": "1. The method might be too expensive. The running time cost should be provided\n\n2. What is the value of N in Algorithm 1?\n\n3. The evaluation does not have generation task for example label-condition generation, text2image generation. This hinders the evaluation of the work.\n\n4. There are recent advances in classifier-free guidance but there is no discussion or comparison in the main paper.\n\n[1] https://arxiv.org/abs/2404.07724 \n\n[2] https://arxiv.org/html/2404.13040v1", "theoretical_claims": "I checked the theoretical claims", "experimental_designs_or_analyses": "Good", "relation_to_broader_scientific_literature": "Provide a new guidance method with optimization on guidance term before doing guidance", "essential_references_not_discussed": "There are recent some advance in classifier-free guidance but does not have the discussion in the main paper.\n\n[1] https://arxiv.org/abs/2404.07724 \n\n[2] https://arxiv.org/html/2404.13040v1", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. Please compare the running time of the algorithm with other CFG methods.\n\n2. What is the value of N in Algorithm 1?\n\n3. The evaluation does not have generation task for example label-condition generation, text2image generation. This hinders the assessment of the work. Please include these tasks in the paper.\n\n4. The authors should include the analysis of the method when combining with other methods such as Interval Guidance, CFG++", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aRsPzBwFhm", "reviewer": "Reviewer_113Q", "summary": "In this paper, the authors formulate the diffusion posterior guidance problem as a variational control problem and propose a novel training-free framework for this problem. They introduce a new algorithm for diffusion guidance and their framework also unifies many existing training-free diffusion guidance algorithms. They have conducted experiments on FFHQ-256 and ImageNet-256 datasets to show superior performance in comparison to several popular baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and helpful feedback. Please see our response below.\n\n> I do find the statement [...] the authors claim in the introduction [...] that prior works which are based on optimal control [...] (all of which are widely studied [1,2,3,4])\n\nOur framework indeed generalizes to non-linear tasks like Style and text guidance. Due to space constraints, please see our response to Reviewer UY5V (first point).\n\n> Moreover, the authors claim to be able to [...] However, the only blind inverse [...] to be inaccurate and not well justified by the evidence provided.\n\nFor the blind deblurring task considered in this work, the forward model y = k * x + n is non-linear since both the kernel k and the underlying signal x are unknown here, making the inverse problem non-linear and challenging. Therefore, we disagree that it is equivalent to solving linear deblurring. Moreover, as highlighted in our common response, we already show that our method can generalize to tasks like Style Guidance, providing further evidence in favor of its generalizability.\n\n> The authors have [...] proposed mostly valid evaluation criteria for the problem at hand [...] performance gain based on FID results in their abstract\n\nWe agree that 1000 samples may produce a biased FID. However, we evaluate all baselines similarly, so we expect similar improvements in FID for more samples. In addition, the reported FID strongly correlates with other more stable metrics, such as LPIPS and KID. Lastly, our choice of metrics and evaluation protocol is primarily based on prior work in the literature like DPS [Chung et al.] and RedDiff [Mardani et al.] and is therefore justified, as also noted by the reviewer.\n\n> The authors choose addition as the aggregation function [...] further justification and clarification on the assumptions when choosing the aggregation function of the control.\n\nWe do regularize u_t via the guidance scale $\\gamma$, compare Eq. 17 in the paper. More specifically, the first term in Eq. 17 regularizes the magnitude of u_t, while the second term encourages a u_t which does not change the magnitude of noise in the guided signal x_t. This brings the guided trajectory close to the unguided trajectory and close to the usual input to the learnt score. Regarding the image manifold, since u_t is only added inside a score function, the terminal cost cannot make the trajectory leave the span of the score function. More importantly, superior empirical results across a range of linear and non-linear inverse problems justify our choice of aggregation function. Lastly, we did not explicitly explore specifying terminal costs that are incompatible with the diffusion prior. We think that this is an interesting future direction and will add this to the Limitations in Section 6 of the revised paper. We hope this clarifies any concerns about our choice of aggregation function.\n\n> The authors mentioned that the Tweedie’s estimate in DPS causes high computational requirement [...]  exempt from the same limitation.\n\nWe think that DPS converges worse because the overall framework assumes that the gradient of the terminal cost is accurate at every step, but it is actually approximated by Tweedie’s estimate. We think that our framework is able to deal with this approximation to the terminal cost better through the additional flexibility in the guidance. We propose rephrasing the corresponding section for the final manuscript to reflect this intuition.\n\n> I have reviewed the appendix [...] So far the qualitative comparison is not comprehensive.\n\nWe include some more qualitative results on FFHQ [here](https://imgur.com/a/2cVdEff). In light of the new results on style guidance we will update the Appendix with more experimental details for these tasks, and additional qualitative results.\n\n> The most prominent concern I have about this paper is the lack of comparison with relevant literature [...] performance than the baselines selected by the authors.\n\nThanks for your feedback and for suggesting other competitive baselines. Based on the reviewers' suggestion, we compare with two additional baselines, MPGD and RB-Modulation, on the non-linear deblur and 4x superresolution tasks for the FFHQ and ImageNet datasets. NDTM mostly outperforms these additional baselines on these tasks. Due to space constraints, please see our response to Reviewer UY5V (Limited Empirical Comparisons).\n\n> The authors choose  T=200,N=15  as the hyperparameters [...] Can the author explain why they can achieve this speedup?\n\nThe configuration T=200, N=15 is used only for the blind inverse problem, which is inherently more challenging and benefits from additional optimization. However, for all other tasks, we use a more efficient configuration (See Table 8), carefully balancing performance and runtime. The runtimes reported in Table 9 correspond to the superresolution task, where T=50, N=5. This setup allows our method to be over 14x faster than DPS on that task.", "claims_and_evidence": "I do find the statement that the authors make about the \"broad generalizability\" of their method to be overclaiming.\n\nIn particular, the authors claim in the introduction that prior works which are based on optimal control only \"focus on a restricted class of control problems\", and their method \"adapts well to diverse downstream tasks\". However, in their experiments, the authors only show evidence on simple inverse problems like deblurring and inpainting. The only non-linear task that the authors explore is non-linear deblurring, and none of the more complicated non-linear task such as style guidance generation, face identity guidance generation or text conditioned generation (all of which are widely studied [1,2,3,4]).\n\nMoreover, the authors claim to be able to solve \"(blind) non-linear inverse problems\". However, the only blind inverse problem that they attempt to solve is linear deblurring. (They assume that there exist a blurring kernel and they would apply linear convolution with this kernel).  As a result, I find the claim about the diversity and the generalizability of the method to be inaccurate and not well justified by the evidence provided.\n\n\n[1] Bansal, et al. Universal Guidance for Diffusion Models. ICLR 2024.\n\n[2] Yu et al. FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model. ICCV 2023.\n\n[3] He et al. Manifold Preserving Guided Diffusion. ICLR 2024.\n\n[4] Rout et al. RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control. ICLR 2025.\n\n[5] Ye et al. TFG: Unified Training-Free Guidance for Diffusion Models. NeurIPS 2024.", "methods_and_evaluation_criteria": "The authors have provided sufficient justification and proposed mostly valid evaluation criteria for the problem at hand. However, since the evaluations are conducted with only 1000 samples, FID is not an appropriate metric for these experiments (FID is highly biased when using a small number of samples [1]). Given that the authors also have included the KID results, which is the more appropriate metric to be used, I am only mentioning this because the authors have emphasized their performance gain based on FID results in their abstract.\n\n[1] Binkowski et al. Demystifying MMD GANs. ICLR 2018.", "theoretical_claims": "I have two major concerns regarding the theoretical aspect of this paper.\n\n1. The authors choose addition as the aggregation function to incorporate control for calculating the posterior mean. However, this design choice is not justified at all. As a matter of fact, without any constraint on $u_t$, the diffusion model may not even be well defined on $x_t + \\gamma u_t$ because all noisy samples are concentrated on certain intermediate manifolds during the diffusion process [1]. It would be great if the authors can provide further justification and clarification on the assumptions when choosing the aggregation function of the control.\n\n2. The authors mentioned that the Tweedie’s estimate in DPS causes high computational requirement and high sensitivity to the hyperparameters. However, in the proposed approach, the authors also use the Tweedie’s estimate for $\\hat{x}_0$. I wonder why the authors consider their method to be exempt from the same limitation.\n\n[1] Chung et al. Improving diffusion models for inverse problems using manifold constraints. NeurIPS 2022.", "experimental_designs_or_analyses": "My concerns about the experimental design are elaborated in section \"Methods And Evaluation Criteria\" and \"Essential References Not Discussed\".", "relation_to_broader_scientific_literature": "One of the biggest strength of this paper is that the authors do provide a unified framework that generalizes diffusion posterior sampling method to an optimal control inspired formulation. Solving inverse problems with training-free diffusion guidance is also a vibrant and important research field as it is closely related to imaging processing, broader signal processing and many medical applications.", "essential_references_not_discussed": "The most prominent concern I have about this paper is the lack of comparison with relevant literature. As I have mentioned earlier in this review, the authors fail to mention and/or compare a large portion of related works in training-free diffusion guidance, many of which achieves better performance than the baselines selected by the authors. In particular, here is a list of papers that the authors should consider comparing to:\n\n[1] Bansal, et al. Universal Guidance for Diffusion Models. ICLR 2024.\n\n[2] Yu et al. FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model. ICCV 2023.\n\n[3] He et al. Manifold Preserving Guided Diffusion. ICLR 2024.\n\n[4] Rout et al. RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control. ICLR 2025. (The authors have mentioned this work in their paper but failed to compare with this method)\n\n[5] Ye et al. TFG: Unified Training-Free Guidance for Diffusion Models. NeurIPS 2024.\n\n[6] Chung et al. Improving diffusion models for inverse problems using manifold constraints. NeurIPS 2022.", "other_strengths_and_weaknesses": "I have elaborated the strengths and weaknesses of this paper in the previous sections.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The authors choose $T=200,N=15$ as the hyperparameters in their experiments. So in total there will be $200\\times(15+1)=3200$ diffusion NFE required for each sample. However, their method is significantly faster than DPS, which only requires $1000$ NFE. Can the author explain why they can achieve this speedup?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AAIJ1JgUmc", "reviewer": "Reviewer_UY5V", "summary": "The paper introduces a framework called Diffusion Trajectory Matching (DTM) for guiding diffusion models without requiring retraining. This approach is rooted in variational inference and optimal control, allowing for guidance by optimizing control signals based on terminal costs. The proposed method, Non-linear Diffusion Trajectory Matching, integrates with existing samplers like DDIM and targets performances on linear and non-linear inverse problems by optimizing diffusion trajectories toward a desired outcome, showing improvements in metrics like FID over some baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Please find our responses below.\n\n> The evaluation tasks make sense but the comparisons [...] lack [...] more recent methods. . Please refer to Essential References Not Discussed.[...] authors did not discuss or compare, such as [1-8].\n\nThis was also a common point from Reviewers DxJn, 113Q so we address it jointly here. \n\n**Re. Generalization to other tasks**. Our framework indeed generalizes to non-linear tasks such as those pointed out by the reviewers. We demonstrate this by simply adapting the terminal cost in our method for Style and text-conditioned guidance. In more detail:\n\n**Style Guidance**: Following the experimental setup in MPGD [He et al.], we evaluate our method on Style Guidance using Stable Diffusion (see [Figure 1](https://imgur.com/a/wEDqoaR)). We further quantitatively compare our method on 1k (reference images, prompts) with FreeDOM [Yu et al.] and MPGD in [Table 1](https://imgur.com/a/jTn0hQ2). Our quantitative and qualitative results suggest that NDTM outperforms competing baselines on prompt (CLIP score) and style adherence (Style Score) while exhibiting better perceptual quality.\n\n**Text Guidance with CLIP**: Following the experimental setup in MPGD, we present qualitative results on text-conditional generation on the CelebA (see [Figure 2](https://imgur.com/a/5jlnDLo). NDTM can be used to generate samples that adhere well to simple and complex prompts.\n\nTherefore, we emphasize that our method can be applied to more general guidance scenarios. We will include additional qualitative results for these tasks with their experimental settings in a revised version of our paper. Note that we selected the best hyperparameters for the selected baselines, just like for all other experiments.\n\n**Re. Limited Empirical Comparisons**. Based on the suggestion from Reviewer 113Q, we compare with two additional baselines, MPGD and RB-Modulation [Rout et al.], on the non-linear deblur and 4x superresolution tasks for the FFHQ and ImageNet datasets. NDTM outperforms these additional baselines on these tasks. See [Table 2](https://imgur.com/a/Y8Mlvw0).\n\nWe also thank the reviewer for the interesting pointers, which we will include in the related work section for broader context. However, they primarily focus on discrete diffusion models or finetuning with human preference data—directions that differ from the scope and goals of our paper. In detail:\n\n**Re. Discrete Diffusion Models.** Our work focuses on continuous state space diffusion models (see Section 2) and assumes a differentiable terminal cost (see line 120, Right Column in the main text). We think that optimal control provides an interesting approach to guiding discrete diffusion models and leave this open to future work.\n\n**Re. Finetuning with human preference data.** In this work, we only work with training-free methods for conditional generation tasks and not with methods that finetune diffusion models with additional human preference data. Therefore, comparisons with baselines like Diffusion-DPO are outside the scope of this work.\n\n> The paper does not provide code for verification\n\nWe will make our code publicly available upon acceptance. Please refer to the pseudocode in Algorithm 1 in the main text and the corresponding hyperparameters in Tables 4,5 and 6 in the Appendix.\n\n> For the comparison with DPS [...]  should also be selected from a space.\n\nWe tuned all baselines (including DPS) for the best performance for all tasks and datasets (see Appendix B.2).\n\n> While the paper discusses non-linear control [...] of non-linear dynamics.\n\n We respectfully disagree, as there seems to be a misunderstanding: We both consider non-linear inverse problems and allow for non-linear control. In detail, while we do update the state $x_t$ additively with the control $u_t$, the updated state ($x_t + \\gamma u_t$) is directly injected in the diffusion denoiser (modeled as a non-linear neural network). Therefore, the guidance process indeed depends non-linearly on the control. We acknowledge that to update the state $x_t$ with the control $u_t$, there could be multiple ways to achieve this. However, we don’t explore the full space of such possible combinations due to space and time constraints which we also highlight in line 175, Right column.\n\n> The method assumes differentiable objectives [...] generative tasks.\n\nWe clearly state our assumptions of a differentiable terminal cost in Line 120, Right Column in the main text. Therefore, while the argument that our method may not be ideal for all types of generative tasks is correct, we never claim this in our work. We agree that extending our method to discrete diffusion models and discrete terminal costs can be an interesting direction for further work and will clarify this point in more detail in Section 6 of our revised paper.\n\n> The paper lacks Impact Statements, which is required.\n\nWe have provided the impact statement at line 435 (Right Side) in the main text.", "claims_and_evidence": "The claims made in the submission are generally right.", "methods_and_evaluation_criteria": "The evaluation tasks make sense but the comparisons are limited.", "theoretical_claims": "The paper doesn't include theorems. The Proofs section in the submission is essentially formulations instead of rigorous math proofs.", "experimental_designs_or_analyses": "There lack experimental comparisons with more recent methods. Please refer to Essential References Not Discussed.", "relation_to_broader_scientific_literature": "The paper discussed the relation of its idea with Classifier Guidance in Diffusion Models and Optimal Control. The key contributions of the paper relate well to the broader scientific literature by extending the capabilities of diffusion models in inverse problem-solving.", "essential_references_not_discussed": "The paper lacks a comprehensive literature review. There are many guidance methods in diffusion which the authors did not discuss or compare, such as [1-8].\n\n[1] Unlocking guidance for discrete state-space diffusion and flow models.\n\n[2] Monte carlo guided diffusion for bayesian linear inverse problems.\n\n[3] Amortizing intractable inference in diffusion models for vision, language, and control\n\n[4] Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding\n\n[5] Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction\n\n[6] Alignment without Over-optimization: Training-Free Solution for Diffusion Models\n\n[7] Diffusion Model Alignment Using Direct Preference Optimization\n\n[8] DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling", "other_strengths_and_weaknesses": "Strengths:\n1. The integration of variational control into the guidance of diffusion models is well-founded.\n2. Demonstrated improvements on tasks such as non-linear and blind inverse problems show broader practical applications.\n\nWeaknesses:\n1. The paper does not provide code for verification.\n2. For the comparison with DPS, the paper seems to fix the hyperparameters. Given that the method itself's performance heavily depends on the correct setting of hyperparameters such as guidance weight and terminal cost weight, hyperparameters of DPS should also be selected from a space.\n3. The paper lacks a comprehensive literature review. There are many training-free guidance methods in diffusion which the authors did not discuss or compare. Please refer to Essential References Not Discussed.\n4. While the paper discusses non-linear control benefits, comparisons primarily focus on linear control setups, possibly overlooking the full spectrum of non-linear dynamics.\n5. The method assumes differentiable objectives, which may not be applicable or ideal for all types of generative tasks.\n6. The paper lacks Impact Statements, which is required.", "other_comments_or_suggestions": "Please refer to above.", "questions_for_authors": "Please refer to above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J220DW0sgW", "reviewer": "Reviewer_amRw", "summary": "This paper introduces DTM - Diffusion Trajectory Matchning - a novel and general guidance approach for generic diffusion models. The idea is to add a guidance vector at each time point, u_t, such that it serves the given measurements while also maikg sure that the original diffusion trajectory is not deviated much. The vector u_t is found by minimizing a cost function that takesthe above two forces into account, and then it is added as the guidance for the current stage. When simplified, this approach amounts to a guidance that is similar to DPS (but avoiding the derivative through the denoiser) , but with a regularization that forces a proximity between the means of the conditional probability means (\\mu(x_t) versus \\mu(xt+\\gamma*u_t)). This last result leads to a non-linear computaiton of u_t, termed NDTM (Non-Linear Diffusion Trajectory Matching). A further simplified version of this is derived for DDIM, penalyzing for the length of u_t, and the difference between the two denoiser outputs. Various experiments demonstrate the superiority of this approach over the alternative methods in handling inverse problems, linear and non-linear ones.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback about experiment design and readability. We address specific questions below:\n\n> Of these three, #2 complicates things and makes the method non-linear and more complicated for optimization. Therefore, an ablation that shows the effect of excluding #2 is necessary. Perhaps #1 would be sufficient to regularize the guidance. Furthyeremore, for linear inverse problems, ut will have a closed form solution of #2 is omitted.\n\nWe would like to thank the reviewer for their insight. We agree that such an ablation could be useful to study the impact of different terms in the loss function in Eq. 17. Empirically, we find the choice of weighting to be largely task-specific (see Tables 7 and 8).", "claims_and_evidence": "Excellent paper", "methods_and_evaluation_criteria": "Perfect.", "theoretical_claims": "All are correct.", "experimental_designs_or_analyses": "Very well designed experiments", "relation_to_broader_scientific_literature": "Explains well the surrounding papers, including stochastric control methods that inspired this work.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Nothing to add. This paper is pleasant to read.", "other_comments_or_suggestions": "Looking at Eq. (17), there are 3 forces that are involved in the creation of ut: \n1. Forcing the norm of this vector to be small (||ut||_2)\n2. Forcing the trajectory not to deviate much by forcing the two denoiising results to be close by, and \n3. Forcing the measurements in some sort of projection step on the denoised image. \n\nOf these three, #2 complicates things and makes the method non-linear and more complicated for optimization. Therefore, an ablation that shows the effect of excluding #2 is necessary. Perhaps #1 would be sufficient to regularize the guidance. Furthyeremore, for linear inverse problems, ut will have a closed form solution of #2 is omitted.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to model guided diffusion dynamics as a Markov chain and develops diffusion trajectory matching for training-free guidance. The paper originally received 2xWeakReject, 1xWeakAccept, and 1xAccept. The main concerns include running time, insufficient evaluations, unclear statements, etc. The authors have provided rebuttals and addressed most concerns of reviewers. Afterward, one reviewer raises the rating. Reviewer UY5V does not respond and keeps negative. However, the authors have clarified the concerns of Reviewer UY5V in the rebuttal. Considering the rebuttal and discussions from all reviewers, ACs recommend accepting this paper. The authors are suggested to carefully revise the paper and incorporate newly conducted experiments according to the comments and discussions.
Decision: Accept (poster)