Constrained Diffusion with Trust Sampling
Authors: William Huang
Yifeng Jiang
Tom Van Wouwe
Karen Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
guidance
image generation
human motion
Primary area: diffusion_based_models
TL;DR: A new optimization-based training-free guided diffusion method, applicable to drastically different domains such as image and human motion generation

Abstract: Diffusion models have demonstrated significant promise in various generative tasks; however, they often struggle to satisfy challenging constraints. Our approach addresses this limitation by rethinking training-free loss-guided diffusion from an optimization perspective. We formulate a series of constrained optimizations throughout the inference process of a diffusion model. In each optimization, we allow the sample to take multiple steps along the gradient of the proxy constraint function until we can no longer trust the proxy, according to the variance at each diffusion level. Additionally, we estimate the state manifold of diffusion model to allow for early termination when the sample starts to wander away from the state manifold at each diffusion step. Trust sampling effectively balances between following the unconditional diffusion model and adhering to the loss guidance, enabling more flexible and accurate constrained generation.  We demonstrate the efficacy of our method through extensive experiments on complex tasks, and in drastically different domains of images and 3D motion generation, showing significant improvements over existing methods in terms of generation quality.  Our implementation is available at https://github.com/will-s-h/trust-sampling.

Constrained Diffusion with Trust Sampling
William Huang Stanford University wi willsh@stanf
Yifeng Jiang Stanford University yif @ettnf edu
Tom Van Wouwe
C. Karen Liu Stanford University karenliu@cs stanford.edu
Stanford University
tvwouse@stanforre edu
Abstract
Diffusion models have demonstrated significant promise in various generative tasks; however, they often struggle to satisfy challenging constraints. Our approach addresses this limitation by rethinking training-free coss-guided diffusion from an optimization perspective. We formulate a series of constrained optimizations throughout the inference process of a diffusion model. In each optimization, we allow the sample to take multiple steps along the gradient of the proxy constraint function until diffusion level. Additionally, we estimate the state manifold of diffusion model allow for early termination when the sample starts wander away from the state manifold at each diffusion step. Trust sampling effectively balances between following the unconditional diffusion model and adhering to the loss guidance, enabling more flexible and accurate constrained generation. We demonstrate the efficacy of our method through extensive experiments on complex tasks, and in dras- tically different domains of images and 3D motion generation, showing significant improvements over existing methods in terms of generation quality. Our implemen- tation is available at htttt::t/g/tteeeee
no longer trust the proxy, according to the variance each
Introduction
Diffusion models are class of generative models that have been highly successful at modeling
complex domains, ranging from the generations of images [3] and videos [24], to 3D geometries
32 54
4 and 3D human motion [52] 550, outperforming other deep generative models, such as
GANs and VAFs (50, I3 23 Originally for unconditional generation, Diffusion models soon became
used for cross-domain conditioned generation, such as text-conditioned image generations 43 [3
and generating human movements from audio [
For more fine-graine conditional generation where the samples need to precisely follow specified constraints, such as generating images following a certain contour, high-level controls like text prompts become insufficient. Guided diffusion has recently emerged to be powerful paradigm variety of such constraints. One category of guided diffusion uses separately trained classifier (as in classifier guidance (13)) or the score of conditional diffusion model classifier-free guidance 21
For new constraints, the classifier or the conditional diffusion model must be retrained 60 42 57
the gradient of loss function representing constraint guidance to achieve conditional diffusion [1] This flexible paradigm allows various constraints to be applied pre-trained diffusion model without compute cost extra training. On this front, since the seminal works of Chung et al. [I]] and Ho et al. [24]. a number of techniques have been proposed to improve the quality of loss-guided diffusion, such better step size design (58), multi-point MCMC
Alternatively.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Image Tasks
Super Resolution
Inpainting
Gaussian Deblur
Human Motion Tasks
Hand Foot Tracking
Obstacle Avoidance
Figure Trust Sampling can be applied to complex constraint problems in drastically different domains.
approximation
incorporation measurement models [46]. Several challenges remain for the current paradigm when trying to apply loss puss-guided diffusion for challenging constraints. For one, performance drops significantly when using a smaller budget of inference computation with fewer neural function evaluations (NFEs) [ [88 The methods also sensitive initialization, where previous evaluations often times take the best of a few generated samples for each constraint input.
In light of these challenges in training-free guided diffusion, we introduce Trust Sampling, novel method that strays from the traditional approach alternating between diffusion steps and loss-guided gradient steps in favor of more general approach, considering each timestep optimization problem. Trust Sampling allows for multiple gradient steps on a proxy constraint function at each diffusion step, while scheduling the termination of the optimization when the proxy cannot trusted anymore. Additionally, Trust Sampling estimates the state manifold of the diffusion model to allow for early termination. if the predicted noise magnitude of the sample exceeds the expected one in each diffusion step. Our framework is ficxible, efficient, and performs well, achieving higher quality across widely different domains (e.g. human motion and images). We demonstrate the generality of Trust Sampling across a large number image tasks (super-resolution, box inpainting, Gaussian deblurring) and motion tasks (root trajectory tracking, hand-foot trajectory tracking, obstacle avoidance, etc.). When compared to existing methods, we find that Trust Sampling satisfies constraints better and achieves higher fidelity.
independent
2 Background
Diffusion models. There are several equivalent formulations for diffusion models used in literature. Here, briefly offer background on the denoising diffusion probabilistic model (DDPM) [22] formulation. Beginning from the data distribution Xo ~ p(x), we can use a variance schedule Bi,..., fr to produce latent variables XI, ... XT through the forward diffusion process g(xx V(Vo,x, (1 0z II), where 0a == IT (1 - 3.). In turn, de-noising model ég be trained by minimizing the following loss function, which is re- reiweiggting of the variational lower bound 22
(()) Ex.x. [lle es (x,, t)]
where Xg ~ pxx), t ~~ Unif(1,.... ,T,, e ~ NO, and x, = Xx20 + -I- aye. The diffusion
model then be sampled in the reverse process, via DDPM 22 model (DDIM) formulation 45
the denoising diffusion implicit
X
(x + V 1- at11 ofco(xe,1) + 022,
(2)

---

where ~ (O, I) in both DDIM and DDPM, whereas or - fixed in DDPM and can be chosen freely in DDIM. X (x,) denotes the predicted X, at timestep t. and can be written as
xx(x)) = _
(xt -
aieoxx 1)).
Voz
Notably, DDPM and DDIM sampling can also be thought of special case of gradient-base based MCMC sampling (or a probability flow, in cases of DDIM without noise), where the goal is to refine the starting sample at each level x, towards maximizing the likelihood x;-1 ~ p(x,-1). In the case of DDPM/DDIM, instead of taking multiple MCMC steps [7 following the score function, only one step taken at each level.
Training-free Guided Diffusion. One important application of Diffusion models is controlled (guided) generation. Instead of sampling from the unconditional data distribution p(x), the goal is to sample from p(x(y) where y is the usually under- specif--e guidance signal. For example, an animator may wish to use an unconditional diffusion model of human motion p(x) to generate motions with constraint that the character's r's right hand reaches to a specific location. Previous works [3 transform the maximization of p(x:ly) at each diffusion level with Bayes' rule:
Vx, p(xl)) = Vx, p(x) + Vx, log (x,),
where note that Vx, log O. Existing algorithms therefore alternate between following the score function of the trained Diffusion Vx, log p(x), and following the guidance gradient Vx, log xx).. However, directly optimizing p(y (x.) is generally intractable ([]] as can be seen by the following probability factorization:
-f p(xu (
p(y|x))= =
(5)
where we used the fact that given Xo. is conditionally independent from x,, In general approximat- ing p(xo|xe) requires many denoising iterations of the Diffusion model, which is impractical when needing to alternate with optimizing p(x)).
To address this difficulty, previous works 58 approximate p(y(x,) with (x))))((x,)) Their observation is that in many practical applications, practitioners do have access to a closed-form differentiable function L(xo, y) that can measure how good clean (predicted sample matches the desired condition y. For example, L can simply be the mean-squared error between target and actual positions of the right hand in our aforementioned animation application. Technically, by defining L(x,, y) such that p(y xx) x exp(-L), p(y (x,) can be maximized by following the gradient direction Vx, y).
ground-truth)
Such frameworks open the door for highly flexible guided diffusion. Using the same unconditional model trained for p(x), we can now plug in various different L for different y during inference time, without having to train additional networks for each possible new y.
3 Trust Sampling: Formulating Guided Diffusion as Optimization
Our work revisits training-free guided Diffusion from the perspective of optimization. Previous works decouple the two terms p(xx) and p(y (x.) in Eq. 4 they use the unconditional Diffusion model to optimize for p(x.), and then took one gradient step of log p(y xx) for the constraint (guidance) term. As our experiment results will demonstrate, single gradient steps for constraints can lead to less optimal samples. With previous works mitigating this issue by carefully selecting the step sizes of Vx, xx log p(y) xx)) (58) or by better approximating ply (x, [47). we explore new direction which leads to a robust practical algorithm across multiple domains and various constraint diffusion tasks. To start with, following the gradients of log p(y| indicates we can reformulate the constraint diffusion problem as an optimization problem:
max p(ylx') subject to x' ~ p(x)),
where we replace x, with x' to signify that the state variable x' can deviate from the Diffusion- predicted x, during this optimization. is important to note that, first, for optimizing p(x;), we still follow standard diffusion inference given its widespread empirical success in multiple domains, and
3

---

second, we constrain x' to stay in the distribution of all possible x, at diffusion level t, as to not create train-test discrepancy for the base diffusion model. This optimization formulation opens the door for more more flexibility in algorithm design, as we are no longer limited to taking only one gradient step.
3.1 Trust Schedules: Termination Criteria of Optimization
Our key improvement of this work is the use of iterative gradient-based optimization to solve Eq. While only taking one single gradient step proves be sub-optimal, as we will demonstrate in this section, optimizing until the objective saturates is also not ideal. To see this, recall that p(y x²) generally not tractable, while p(y X (xx) is. As we replace the optimization objective ply xx) with the proxy p(y (xo it is crucial to terminate in time before the proxy becomes poor approximation of the true objective. Formally this relaxation can be written as:
6
min L(xo(x'), subject x' p(x.),
[p(y(x') - p(y| < <d,
where readers are reminded that minimizing L(8o: ) is equivalent to maximizing p(y [8o), and is a relaxation threshold newly introduced. To reason about the gap d between true and proxy objectives, note that:
p(ylx') = -f p(xoo>'' ((xolx))py/// (y(p))p((((())))))))) (x)) = Exo (xo)]
xg
p(y|2)) = f(xo) = ((xx)
where we use f(-) as a shorthand for exp(-L(-:))) ((-)) While a similar but tedious analysis exist for general multivariant x. for the purpose of practical algorithm design, looking at the special case where x scalar random variable is more intuitive for understanding. Let f" denote the curvature of fx), and inf f" and b = sup f" denote the range of the curvature assuming x has a finite span, we now have f 2 ax² and f both as convex functions. Applying Jensen's 's inequality to both functions:
E f(xo) 2 [xx3
f((xx)) aB(xol?
E
1 bxd
5(x00)
2
1
bE bE|xol? - f(E(xul). (8)
After rearranging this gives:
2 Var(x) S [[f((x)] (E(xx)) S Var(x)).
This indicates, rather intuitively. that the approximation error increases with the variance of As result, we can trust the proxy optimization minx' L(xg(x'), y) more when the variance of x is smaller and the proxy becomes less reliable when the variance is large. Since it is intractable to estimate the true value of the gap d during the course of optimization, we opt to design a trust schedule of maximally allowed gradient iterations that correlated to the variance of x at each diffusion iteration f. In our experiments, we will demonstrate that simple schedules, such as constant function Strust (t) = c, or a linear function Gtrust (t) m t + c, work surprisingly well for the diverse set of tasks we attempted.
3.2 Early Termination Using State Manifold Boundaries
The previous section reformulates constrained guided diffusion as gradient-based optimization, with our proposed algorithm designed to timely terminate the iterations based on the trustworthiness of the proxy objective. Equations 6 and aaddiiooally require us characterize the space that a forward sample x, can possibly visit at diffusion level f, so that we can ensure, during inference time, that x' will not leave the state manifold where the base model is trained on. In practice, the robustness of diffusion models can produce valid samples even if the input is slightly outside of the state manifold, allowing the constraint ~ p(x;) to be relaxed. However, stepping outside of the state manifolds might require more unnecessary "corrective" steps, affecting the run-time performance. To speed up the computation during inference time, we describe method for early termination of the optimization when the sample leaves the estimated boundary of the state manifold at each diffusion step.

---

We leverage the boundaries of a Diffusion model's intermediate state manifolds Mr,., which we define per diffusion timestep as the manifold on which a diffusion model has likely seen training data from with probability 2 6:
M1,5 X / (xxx)x0))(xx)))xx 21 21-6%
(10)
Given sufficiently small 5 and sufficiently well-trained diffusion model, the idea is that any X₁ E MM. will converge to some point Xo in the original data distribution p(xo). As such, the optimization problem from Eq. X becomes:
min L(xo(x').y) subject to x' E &Me,. ((yyx') - p(y|)())))
(11)
x'
By definition, Mr. is larger manifold when is larger, meaning it gradually shrinks to true data manifold during diffusion inference. Nevertheless, Mr,5 would be challenging to compute closed- form given the unknown true data distribution p(xp). Our observation is that in all formulations of Diffusson models, we do have access to the model's predicted noise E. For a particular x', the ideal value for Eg (x', , t) is
co Ex(x't) tx(x x
vopXo 0t
Xx'
p(xx) Exp
(12)
Ot
If x' is within the state manifold boundary, the integrand <T-a; correspond to a multivariant Gaussian (O, This implies that we can estimate the boundary of
Yor for each sample of Xo should
M1,5 with |xx, (xxxx)))ll When ||ea(x', t)) || is far away from zero, VI-ai is unlikely to be sampled from ((, I). Consequently, x' likely to outside the state manifold at the current diffusion step. In practice, set such threshold émar by observing the approximate average ||eg (x,, t)]l across several unconstrained samples running the base Diffusion model.
x' x'=vaxXo
3.3 Algorithm
Comparing with standard DDIM sampling, our Trust Sampling algorithm (Algorithm T takes multiple gradient steps of constraint guidance up to a maximum of Jr, which denotes a max-iteration according to the trust schedule, (t). We experiment with different linear schedules, detailed in the Experiments section, show the positive impacts of our algorithm on the quality of generated data. The inner optimization loop will also be terminated by the condition when the magnitude of the predicted noise e being larger than max- Following Yang et (58), we normalize the gradient for numerical stability. w is constant step size which we keep either as 0.5 or 1.0 for each specific task.
Algorithm 1: Trust Sampling with DDIM
Require: (0, I), T, observation y, trust schedule Gtrust (1), norm upper bound émax: guidance weight w
for t T,..., do
[g /0 /1 a--1 10-11-fco (x.)
xi-1.j 0
Jr + Perust (t)
while j < Jj and ((00 ||| < emax do
V a;
L(
j+ j+1
end
C, N(O I)
10
C:-1 ./-1 OtEr
end
use the mean-squared value over all constraint violations to compose I, in the case of equality constraints. However, we need to make an to handle inequality
Adapting Inequality Constraints.
constraints. In the case of an inequality constraint (4(x) > a, we choose formulate I, = max(0, (x))). We then compose L as the mean-squared value over all Li.
5

---

4
Related Work
Our work is most closely related to zero-shot guided Diffusion methods for general loss functions. The seminal works of [ and [24] introduced a method that alternates between taking one denoising step of the unconditional base diffusion model maximize data distribution and taking one constraint gradient step to guide the model for conditional sample generation. This approach effectively balances data fidelity and conditional alignment. DSG [58] enhanced []] by normalizing gradients in the constraint guidance term and implementing step size schedule inspired by Spherical Gaussians. LGD-MC [47) addressed the inherent approximation errors in DPS by using multiple samples instead of single point, which provided a better approximation of the guidance loss. Manifold Constrained Gradient (MCG) [10) and Manifold Preserving Guided Diffusion (MPGD) [19] use projections on the constraint gradient and predicted de-noised sample respectively to leverage the manifold hypothesis for better constraint following. In contrast, our work explores improving this paradigm using iterative wradient-based optimization.
Various methods have been developed specifically for guided diffusion of image restoration. RED- Diff [36] extends the principles of Regularization by Denoising (RED) for image noise removal 40 to stochastic setting, offering variational perspective on solving inverse problems with diffusion models. Techniques such as [T0] [27 55 14 49] assume linear distortion models and utilize the mcasurement operator matrix to improve guidance for image restoration. To handle non-linear distortion models, approaches like [41] and (56) have been proposed. These methods can accommodate complex distortion but require specialized initialization schemes, which limits their general applicability. In contrast, our approach initializes from the standard unit Gaussian, ensuring broader applicability in general tasks. Similarly, IIGDM [46] addresses inverse problems for image restoration with diffusion, but it is confined to certain loss types, while RePaint [31 enhances diffusion-based image in-painting by repeating crucial diffusion steps to improve fidelity.
Recent advancements like (59] and 6 tackle guided diffusion tasks based on conditional models, with conditions including textual information. FreeDOM (59) additionally adopts an energy-based framework and generalizes the repeating strategy found in RePaint [3]] with a novel time travel strategy. DiAPIR [62] balances the data prior term from the unconditional diffusion model with the constraint term from measurement loss to improve image restoration tasks.
Other methods adopt additional training for controlled diffusion. Ambient Diffusion Posterior Sam- pling [ builds upon DPS [I] by training the base model on linearly corrupted data. A8] learns a score function for the noise distribution, specifically targeting structured noise in images. ControlNet [60] and OmniControl (57) train additional Diffusion branches to process input constraints and conditions, achieving notable results in image or motion domains. DreamBooth (42) fine-tunes base diffusion model to place subjects in different backgrounds using few images, demonstrating versa- tility in content generation. Other notable related works include (15). which focuses on composing multiple difffusion models. The proposed MCMC framework replaces simple gradient addition with more robust iterative optimization process, similar to our framework for solving guided diffusion. D-PNP [77 reformulates diffusion prior for various guidance tasks but has been observed to struggle with more complex diffusion models, such as those trained on ImageNet
5
Experiments
We evaluate our method on two drastically different domains: images and 3D human motion. In both domains, we compare against recent zero-shot guided diffusion algorithms for solving general constraint diffusion: DPS [T. DSG [58], and LGD-MC 47].
5.1 Image Experiments
Tasks. We evaluate our method on three challenging image restoration problems: Super-resolution, Box Inpainting, and Gaussian Deblurring. These common linear inverse problems are standard across DPS [II), DSG [58], and LGD-MC [7]; we note that in this paper, we do not inject noise into the initial observations. Each of these image restoration problems can be thought of as a constraint satisfaction problem, where the constraint is that the generated picture appear the same as the source image upon applying the particular distortion. Distortion for these problems, respectively, was performed via (i) bicubic downsampling by 4x, (ii) randomly masking 128 x 128 square region

---

SR (x4)
Inpaint (box)
Deblur (Gauss)
Methods
FID 4
LPIPS
FID 4
LPIPS
FID 4
LPIPS +
DPS
29.48
0.212
20.19
0.140
23.59
0.195
DPS+DSG
27.06
0.193
18.92
0.137
24.06
0.194
LGD-MC (n 10)
29.59
0.212
20.15
0.141
27.38
0.229
LGD-MC (n = 100)
29.54
0.212
20.13
0.140
27.23
0.228
Trust (ours)
16.99
0.156
15.28
0.141
21.19
0.176
Table Quantitative evaluation (FID, LPIPS) solving linear inverse problems on 1000 validation images of FFHQ 256 x 256. Bold: best, red: worst.
SR (x4)
Inpaint (box)
Deblur (Gauss)
Methods
FID L
LPIPS 4
FID |
LPIPS
FID
LPIPS 4
DPS
111.53
0.353
142.03
0.282
152.57
0.442
DPS+DSG
148.53
0.438
115.90
0.247
145.64
0.406
LGD-MC (n 10)
110.36
0.353
142.77
0.280
142.33
0,424
LGD-MC (n = 100)
108.00
0.349
131.23
0.282
152.53
0.444
Trust (ours)
55.24
0.236
99.87
0.210
69.49
0.266
Table Quantitative evaluation (FID, LPIPS) of solving linear inverse problems on 100 validation images of ImageNet 256 x 256. Bold: best, red: worst.
(sampled uniformly within 16 pixel margin of each side), and (iii) Gaussian blur kernel of size 61 x 61 with standard deviation 3.0. We experimented on two datasets: FFHQ 256 x 256 (26] and ImageNet 256 256 12] on 100 validation images each given our limited compute access. For fair comparison between methods, we used the same pretrained unconditional diffusion models across methods for FFHQ [T] and ImageNet [33 following previous works. Quantitative evaluation of images is performed with two widely used metrics for image perception: Fréchet Inception Distance and Learned Perceptual Image Patch Similarity (LPIPS) [61]
(FID) 20
Results. Quantitative evaluation results can be seen in Tables and 2 Our method outperforms diffusion model baselines by a significant margin across all three image tasks on both FID and LPIPS and both FFHQ and ImageNet. Qualitative results Sampling shows an ability to adhere to the original down-sampled image better, even recovering text much better. box inpainting. Trust Sampling fills in the box with realistic output: for example, the eyes in the human faces generated on the right of Fig. 2 are much more natural.
be seen in Fig. 2 In super-resolution, Trust
Monurennet
DPS
DPS-DSG
LGD-MC
Trust (ours) Ground Trath
Momuremeat
DPS
DPS-DSG
LGD MC
Trast (surs) Ground Trunh
Figure 2: Results on solving linear inverse problems. The left shows examples of box inpainting: the right shows examples of super-resolution.
7

---

root tracking
right hand & left foot tracking
Methods
FID 4
Diversity
Const. [m] +
FID
Diversity +
Const. [m]
DPS
542.8
23.8
0.13
604.7
22.5
0.12
DPS+DSG
715.1
25.0
0,022
865.5
24.0
0.035
LGD-MC (n 10)
578.6
21.6
0,031
715.2
22.6
0.056
LGD-MC (n = 100)
579.3
22.6
0.006
731.6
23.1
0.052
Trust (ours)
561.6
21.5
0.026
694.1
20.4
0.038
GT
17.3
17.3
Table 3: Evaluation of FID, Diversity, and Constraint Violation in meters for motion tasks: root tracking and right hand & left foot tracking. Bold: best, red: worst. Computational budget for all methods 1000 NFEs.
5.2 Human Motion Experiments
Unconditional Motion Diffusion Model. For all tasks we use the same unconditional diffusion
model, which we trained on the AMASS [33 2 dataset excluding the following datasets that are used for testing: dance [ HUMAN4D 9 and
28, [6] 8. 7 30 37
34
22
[53
25
51
44
3
Weizmann [35]. The architecture is adapted version of the EDGE motion model removed the branches handling conditions.
52 where we
Metries. We evaluate DPS, DPS+DSG, LGD-MC, and Trust on several constrained motion genera- tion tasks. We train an autoencoder and use the encoder as a feature extractor for motion clips, to allow for calculation of motion realism and diversity metrics 38]. We use the following metrics to evaluate performance:
FID: We extract features using the aforemmentioned encoder and calculate FID between different methods vs. ground-truth, as in Action2MMotion [8].
Diversity: We extract features and calculate the diversity metric in Action2Motion [18 for the generated and ground truth motions. A result is claimed better than others if its score is closer to the score of the ground truth.
Constraint Violation: A task-dependent metric that describes how well the generated motion adheres to the provided constraints.
tasks where we have ground truth motions from the test dataset: root trajectory tracking and right hand & left foot trajectory tracking. Here the diffusion model should be guided to generate natural human movements that closely follow specified root motions hand/foot motions. Note that the generations do not need to match the ground-truth motions due to under-speciffiation of the constraints; we are only using them for generating the control constraints which are guaranteed to be physically feasible for human movements. Specifically, we randomly select total of 1000 slices from the mentioned three test sets, and we extract their root motion and right hand and left ankle motion as constraint signals for the respective tasks. Note also that the observation mapping, from full motion states to the constraint signals, is highly non- non-linear in the hand/foot tracking task. This is because the full motion state of Diffusion only uses local joint rotations, but the hand/foot trajectory defined in the global Cartesian space (see EDGE [2 for more details).
Tasks. We first evaluate
Results. Our method strikes the best balance to matching the constraint without sacrificing realism nor diversity. DPS has the best FID score closely followed by ours. However, this comes large cost for DPS that violates the constraints. DSG satisfies constraints slightly better than our method, but sacrifices both diversity and realism significantly. Our method outperforms DPS and DSG on the Diversity score for both root tracking and right hand & left foot tracking. While LGD-MC balances fidelity and constraint following better, still has worse fidelity than Trust and struggles with harder tasks such as right hand & foot tracking. Note that for these tasks the constraint metric the root-mean-square tracking error in meter. The difference between ours, DSG, and LGD-MC are hardly noticeable when performing visual comparison between the generated motions, especially for root tracking. Visualizations that support these observations are Appendix D, Fig. 6 but are best viewed in the supplementary video.
8

---

More Challenging Tasks. We further experimented our method with more difficult tasks such as sparse spatio-temporal constraints, inequality and highly non-linear constraints, and compositing multiple constraints. This is in drastic contrast to the image tasks, where single "dense" (closer to being fully-specified) constraint must be satisfied. We designed the following tasks with additional two composite constraints each-a translation constraint on the initial and the final frames.
Obstacle Avoidance: We add an inequality constraint to avoid penetration between any joint and three pseudo-randomly placed obstacle spheres.
Jump: We add an inequality constraint the middle frame, to impose that all joints have vertical position that is higher than a selected value between 0.6 m and 1.0 m.
Angular Momentum: We add an inequality constraint to impose different minimum values for the average angular momentum around a horizontal axis. This serves as a way to control dynamicism of motion. Angular momentum is approximated as: Σ Zim1 V; Pi with v., the relative velocity and position of an end effector (wrists and ankles) with respect to the root.
As quantitative metrics would not be informative in these tasks (for example, it is not reasonable to compute the distributional distance between ground-truth -ounh test set and jumping motions), we focus on qualitative demonstrations. We consistently found that for easier inequality constraints (e.g. lower jumping heights) all methods could match the constraints. Howver, our method was more robust when constraints became harder, while DSG sacrificed physical realism and DPS violated the constraints. See Fig. 6 and supplemental videos for more details on these observations.
5.3 Ablations
To examine the influence of Trust sampling, performed ablations the same three image tasks on FFHQ. In addition to FID and LPIPS, we look at the number of neural function evaluations (NFEs) implementation-agneenn
metric of efficiency. In our case, NFEs is the number of times pass through the pretrained model occurs.
Trust Scheduling. We decouple just the trust schedule and do not use state manifold estimates for this part. The results (Table 4 show that our method is not sensitive to scheduling parameters as all schedules still outperform the DPS and DSG baselines on all three image tasks by significant margins. Within the different schedules, we see that linear schedules with non-zero slope (i.e. non-constant schedules) typically outperform constant schedules. This aligns with our notion of trust, as earlier diffusion steps tend to be noiser and therefore the proxy constraint function is less trustworthy, so is less productive to take gradient steps at earlier times. Although linear trust schedule is better than constant schedules, the results indicate the best slope dependent of the task and NFEs.
Fewer NFEs. Table 4 also shows when decreasing NFEs from 1000 (same as baselines) to 600, the performance of our method barely drops and are still significantly better than baselines. To control the desired number of NFEs (1000 or 600 in our experiments), we choose a few combinations of the slope and the offset of the trust schedule (t) the desired number of NFEs, where T' is the total number of diffusion iterations.
+ c, such that E Itrust (t) equals
Manifold Boundary Estimates. We examined the effect of using manifold boundary estimates on the image tasks on FFHQ and ImageNet. We compare the effect of manifold boundary estimates when added to the trust schedule, as compared to only trust scheduling. Table shows the results of using manifold boundary estimates. The use of manifold boundary reduces the needed NFEs by 10-20% without any substantial loss in quality, resulting in better compute efficiency. This performance boost evidently robust across image task, dataset, and NFEs. Table 5 shows that if instead of adopting manifold boundary, we want to achieve the same NFE save by tuning the start and end points the linear schedule, model quality can suffer. Table shows the effect of varying <max- We observe that max generally has an acceptable range (e.g. 440-442 for FFHQ Super Resolution (4x)), within which performance varies only slightly. For the motion tasks we did not find a significant effect when introducing manifold boundary estimates.

---

parameters
SR (x4)
Inpaint (box)
Deblur (Gauss)
Total NFEs
Start
End
FID 4
LPIPS 4
FID
LPIPS 4
FID
LPIPS
1000
4
4
36.95
0.150
34.72
0.146
47.25
0.179
1000
6
35.73
0.152
32.63
0.145
45.56
0.173
1000
8
36.26
0.156
35.08
0.148
42.98
0.173
600
2
45.01
0.153
44.22
0.178
57.12
0.199
600
3
41.56
0.159
44.81
0.178
54.11
0.195
600
4
34.94
0.149
38.70
0.151
48.94
0.181
baseline (DPS)
64.66
0.230
51.25
0.176
60.91
0.226
baseline (DSG)
60.23
0.214
58.30
0.179
59.59
0.212
Table 4: Trust scheduling ablation study NFEs and different trust schedules. Metrics calculated on linear inverse problems on 100 validation images of FFHQ 256 x 256. "Start" and "End" indicate the
boundary conditions of the trust schedule: Jerust (1) Start and Strust (T) End. Bold: best among
same NFEs, underline: second best among same NFEs.
parameters
SR (x4)
Inpaint (box)
Deblur (Gauss)
Bound
Start, End
NFEs
FID 4.
LPIPS 4
NFEs
FID J.
LPIPS
NFEs
FID 4
LPIPS
0,3
500
42.31
0.160
500
48.85
0.186
500
56.57
0.195
1,3
600
41.56
0.159
600
44.81
0.178
600
54.11
0.195
X
0,
600
34.94
0.149
600
38.70
0.161
600
48.94
0.181
0,
497
37.52
0.150
561
41.87
0.169
498
49.95
0.181
Table FFHQ Manifold boundary ablations. Metries calculated on linear inverse problems on 100 validation images of FFHQ 256 x 256. Bold: best, underline: second best.
parameters
SR (x4)
mma
Start,
End
FID
LPIPS
438.0
15, 15
50.81
0.191
439.0
10, 10
40.90
0.165
440.0
5, 5
35.85
0.153
441.0
4,
35.46
0.149
442.0
4.
36.06
0.150
Table 6: max ablations. Metrics calculated on Super Resolution (4x) on 100 validation images of FFHQ 256 256. To isolate purely the effect of mmax while keeping the number of NFEs comparable, constant linear schedules were chosen so that the number of NFEs was close to 1,000. Bold: best.
6 Conclusion
We introduce trust sampling, a novel and effective method for guided diffusion, addressing the current limitations of meeting challenging constraints. By framing each diffusion step as independent optimization problem with principled trust schedules, our approach ensures higher fidelity across diverse tasks. Extensive experiments in image super-resolution, inpainting. deblurring, and various human motion control tasks demonstrate the superior generation quality achieved by our method.
Our findings indicate that trust sampling not only enhances performance but also offers a flexible and generalizable framework for future advancements constrained diffusion-based modeling. To further improve generation quality, future research should adopt holistic approach by incorporating additional concepts from traditional numerical optimization into this framework, beyond just the termination criterion. This includes techniques such as step size line search and fast approximation of higher-order derivatives. Morcover, automating the setting of hcuristic parameters, which are currently manually adjusted for each base diffusion model, would be beneficial.
10

---

References
[1] Asad Aali, Giannis Daras, Brett Levac. Sidharth Kumar, Alexandros G Dimakis, and Jonathan I Tamir. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data, arxiv preprint arXiv:2403.08728, 2024.
[2] Advanced Computing Center for the Arts and Design. ACCAD MoCap Dataset. URL https: osu. edu/ r
and datta
[3] Ijaz Akhter and Michael J. Black. Pose-conditioned joint angle limits for 3D human pose reconstruction. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1446-1455, June
2015, doi: 10.11099CVVR.2205522227555.
[4] Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-20, 2023.
[5] Andreas Aristidou, Ariel Shamir, and Yiorgos Chrysanthou. Digital dance ethnography: Organizing large dance collections. J. Comput, Cult. Herit, 12(4), November 2019. ISSN 1556-4673. doi: 10.1 145/3344383.
URL httpps///// org/ 10. 11 1145/ 1145/3344383 3344383
[6] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild. Soumyadip Sengupta, Micah Goldblum, Jonas Geiping. and Tom Goldstein. Universal guidance for diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black. Dynamic FAUST: Registering human bodies motion. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5573-5582, July 2017. doi: 10. 10.100//VPR.20177
[8] Carnegie Mellon University. CMU MoCap Dataset. URL http:httpc/maaatt cs. was..cmu.ee cmu.edu
[9] Anargyros Chatzitofis, Leonidas Saroglou, Prodromos Boutis, Petros Drakoulis, Nikolaos Zioulis, Shishir Subramanyam, Bart Kevelham, Caecilia Charbonnier, Pablo Cesar, Dimitrios Zarpalas, et al. Human4d: A human-centric multimodal dataset for motions and immersive media. IEEE Access, 8: 17776111176626,, 2020.
[10] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances Neural Information Processing Systems, 35: 25683-25696, 2022.
[11] Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Mare L. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems, 2023.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. leee, 2009.
[13] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
[14] Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations, 2023.
[15] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha
Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International conference on machine learning. pages 1489-85100 10. PMLR. 2023.
[16] Saced Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad Kording, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje. Mo A large multipurpose motion and video dataset, 2020.
[17] Alexandros Graikos, Nikolay Malkin, Nehojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and- play priors. Advances in Neural Information Processing Systems, 35:14715-14728, 2022.
[18] Chuan Guo, Xinxin Zuo, Sen Wang. Shihao Zou, Qingyao Sun, Annan Deng. Minglun Gong, and
Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th
ACM International Conference on Multimedia, MM '20, page 1021-2029, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379885. https://doi. orgo 1111/1114411.1.355 34136665
doi: 10.1145/399/7713441666 URL

---

[19] Yutong He. Naoki Murata, Chich-Hsin Lai. Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji. J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffuusion In Twelfth International Conference on Learning Representations, 2024. URL https::/openrevieww net/forummmiie -d0.mmmdddddd30000mmmm
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Jonathan Ho and Tim Salimans. Classifier-fiee diffusion guidance, 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Asso- ciates, Inc., 2020. URL https::/ httpps://proceeddings wtt.uppp.c//ppp....eeee/pppppppp (ebrrfecbba4f00077faaaaorraabbbb
.df
[23] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022.
[24] Jonathan Ho, Tim Salimans, Alexey Gritsenko. William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03458, 2022.
[25] Ludovic Hoyet, Kenneth Ryall, Rachel McDonnell, and Carol O'Sullivan. Sleight of hand: Perception of finger motion from reduced marker sets. Procedingss of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 13D 12, page 79,86, New York, NY. USA. 2012. ISBN 978145031 doi: URL http:::/toooo org/10. 1145//11911111 2159630
10. 10.1144/21596162221559
[26] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[27] Bahjat Kawar, Michael Elad. Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23661 23606, 2022.
[28] Bio Motion Lab. BMLhandball Motion Capture Database. URL hwt.::tttt://ttt http:////wwwiiettoooonlbbbbbb www.btitoooaaab...oo.oo..o...... ca//
[29] Matthew Loper, Naureen Mahmood, and Michael J. Black. MoSh: Motion and Shape Capture from
Sparse Markers. ACM Trans. Gruph., 33(6), November 2014. doi: 10.1145/2661129 URL
https://doi.. org/ 10111116112222 2661273
[30] Eyes JAPAN Co. Ltd. Eyes Japan MoCap Dataset. URL htRphhttp///ocaaaaataa
[31] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models, 2022.
[32] Shitong Luo and Wei Hu. Difftusion probabilistic models point cloud generation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[33] Naureen Mahmood, Nima Ghorbani, Nikolaus Troje, Gerard Pons-Moll, Michael J. Black. AMASS: Archive of motion capture as surface shapes. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pages 5441-5450, October 2019, doi: 10.110//CC..2019..000
[34] C. Mandery, Ó. Terlemez, M. Do, N. Vahrenkamp, and T. Asfour. The KIT whole-body human motion database. In 2015 International Conference on Advanced Robotics (ICAR), pages 329-336, July 2015. doi: 10.1 1011///A...0....2551111
[35] Christian Mandery, Ömer Terlemez, Martin Do., Nikolaus Vahrenkamp. and Tamim Asfour. Unifying rep- resentations and large-scale whole-body motion databases studying human motion. IEEE Transactions on Robotics, 32(4):796 809, 2016. doi:
[36] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[37] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, and A. Weber. Documentation mocap database HDM05. Technical Report CG-2007-2 Universität Bonn, June 2007.
12

---

[38] Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfir Aberman, Olga Sorkine- Hornung, and Daniel Cohen-Or Modi: Unconditional motion synthesis from diverse data. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 13873-13883, 2023.
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 220::2204000125, 1(2)3, 2022.
[40] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4))1804 1844, 2017.
[41] Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Beyond first-order tweedic: Solving inverse problems using latent diffusion. arXiv preprint arXiv:2312.00 00852, 2023.
[42] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuiigg text-to-image diffusion models for subject-driven generation, 2023.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-lo- image diffusion models with deep language understanding. Advances in newral information processing systems, 35:36479-36494 2022.
[44] L. Sigal, A. Balan, and M. J. Black. HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision, 87(4):4-27, March 2010. doi: 1007/00/11111 0t////11263-000 263-0009-027366
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.
[46] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guiide diffusion models for inverse problems. In International Conference on Learning Representations, 2022.
[47] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yo Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat, Loss-guided diffusion models for plug-and-plan controllable generation. In International Conference on Machine Learning, pages -443932499. PMLR, 2023.
[48] Tristan SW Stevens, Hans van Gorp. Faik C Meral, Junseob Shin, Jason Yu, Luc Robert, and Ruud JG van Sloun. Removing structured noise with diffusion models. arXiv preprint 23t.::233200599 2023.
[49] Matthicu Terris, Thomas Moreau, Nelly Pustelnik, and Julian Tachella. Equivariant plug-and-play image reconstruction. arXiv preprint a32iv.231..01831, 2023.
[50] Guy Tevet. Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXtv preprint ar 220: /4916, 2022.
[51] Matthew Trumble, Andrew Gilbert, Charles Malleson, Adrian Hilton, and John Collomosse. Total capture: 3d human pose estimation fusing video and inertial sensors. In Proceedings of the British Machine Vision Conference (BMVC). pages 14.1-14.13, September 2017. ISBN 1-901725-60-X. doi: 10.5244/C.31.14
Utt./tttss:////// //áx doi. org/10. 10. 5244/C.31. 31.14
[52] Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 448-458, 2023.
[53] Simon Fraser University and National University Singapore. SFU Motion Capture Database. URL http://mocap... cs. sfu. ca/
[54] Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusson models for 3d shape generation. Advances in Neural Information Processing Systems, 35: 10021-10039, 2022.
[55] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.0045 00490, 2022.
[56] Hongjie Wu, Linchao He, Minggin Zhang, Dongdong Chen, Kunming Luo, Mengting Luo. Ji-Zhe Zhou, Hu Chen, and Jiancheng Lv. Diflusson posterior proximal sampling for image restoration. arXiv preprint 2402 16907, 2024.
[57] Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint a0X0022310008550, 2023.
13

---

[58] Lingxiao Yang. Shutong Ding. Yifan Cai, Jingyi Yu, Jingya Wang. and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusion. arXiv preprint 25X2: 2024.
[59] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bemard Ghanem, and Jian Zhang. Freedom: Training- energy- guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23174-23184 2023.
[60] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023.
[61] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable elfectiveness deep features perceptual metric. Proceedings of the IEEE conference on computer vision and pultern recognition, pages 586-595, 2018.
[62] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, Luc Van Gool. Denoising diffusion models for plug-and-ple image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1219-1222 2023.
14

---

A Experiment Details
Image Parameters. The parameters used for all tasks can be found in Table implementing linear schedules, we found the most effective class of trust schedule to be stochastic linear schedules, where the expected values of iteration limits over diffusion time, [E[J)], form an arithmetic sequence.
and the integer iteration limit Ja is determined at runtime by randomly rounding up with probability B[J.]
[.].
Max NFEs
DDIM Steps
Dataset
Task
Start
End
max
1000
200
FFHQ
SR
2
441
1000
200
FFHQ
Inpaint
2
6
442
1000
200
FFHQ
Deblur
2
441
1000
200
ImageNet
SR
0
8
441
1000
200
ImageNet
Inpaint
442
1000
200
ImageNet
Deblur
0
8
441
600
200
FFHQ
SR
441
600
200
FFHQ
Inpaint
0
4
442
600
200
FFHQ
Deblur
0
441
Table 7: Parameters used for all experiments. Start and end refer to the start and end of the stochastic
linear trust schedules.
Motion Parameters. For all motion experiments, we match the computational budget (NFEs) between methods: use 1000 DDIM steps for DPS and DPS+DSG. We spend between 950 and 1000 NFEs for Trust by using 200 DDIM steps with a stochastic stochastic linear schedule using Start 0 and End 8. As mentioned in the experiments, did not find significant effect when introducing manifold boundary estimates for motion and thus there is no max set for the motion experiments.
B Compute Resources
For image tasks, we used pretrained models for FFHQ and ImageNet. We ran inference on an A5000 GPU, which takes roughly I minute to generate an image for FFHQ and 6 minutes to generate an image for ImageNet, due the larger network size. For motion tasks, the diffusion model was trained on single A4000 GPU for approximately 24 hours. Inference does not require large GPU and generating single motion trial, without batching, takes less than 30s.
C Qualitative Samples for Images
Figures 3 and illustrate several examples of Trust sampling Gaussian Deblurring, Box Inpainting, and Super- Super-Resolution respectively on both the FFHQ and ImageNet datasets.
D Qualitative Samples for Motion
Fig. 6 illlstraates several examples of complex motions generated by trust sampling. More results presented in the Supplemental Video.
15

---

Figure 3: Qualitative results for Trust Gaussian Deblurring. The first of images from FFHQ, and the latter two rows of images are from ImageNet.
Figure 4: Qualitative results for Trust on Box Inpainting. The first rows of images are from FFHQ, and the latter two rows of images are from ImageNet.
16

---

Figure 5: Qualitative results for Trust The first two rows of images are from FFHQ, and the latter two rows of images are from ImageNet.
17

---

Hand foot tracking
Jumping
Angular Momentum
tracking
Obstacle Avoidance
Obstacle Avoidance
Angular Momentum
Figure 6: Qualitative results for Trust on different motion tasks. For "Jumping" the horizontal dotted line indicates the required height be cleared.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Abstract and introduction claim more flexible and accurate guided diffusion through our new method of trust sampling, achieving higher quality and efficiency. This accurately reflects the paper's contributions; the method is explained in Section 3 and the results that prove efficiency, quality, and steady performance across multiple domains (i.c. flexible) compared baselines are located in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
19

---

Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We touched on limitations of our work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide short proofs for all newly introduced theoretical results in Section
3. For theoretical results shown by previous works, cite them in Sections 2, 3, and 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
20

---

Justification:
We provide all parameters with regard to trust sampling at each interval in Section 5 and Appendix A, and we fully provide the sampling algorithm needed to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: To our best knowledge, we have provided enough detail to fully recreate our experiments. Our code will be open-sourced at the time of publication, at the GitHub provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips http://wwww.//eeieeee///eettetaa
for more details.
21

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Our paper focuses on an inference technique on top of existing pretrained models. We specify all the parameters used to test new sampling method in Section 5.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: examples.
All metrics we provide calculated sets of 100 images 100+ motion
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
22

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We specify the compute resources in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///orrrpss.ccc//eeee/eeeeee
Answer: [Yes]
Justification: We have verified that all datasets used (FFHQ, ImageNet, AMASS) licensed to allow non-commercial scientific research. We believe there to be a small, but non-zero, risk for the usage of our method for deceptive interactions such as deepfake images, but because our method does not really optimize for recreating particular people in other contexts, we believe the risk low.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Diffusion is core technique that enables many of the applications within Generative AI, many of which have positive benefits on society. Being able to precisely control the generations of diffusion models improve the safety and accuracy of those applications. Although we do not see specific negative impacts of our work, there is always potential for misuse of our technology.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
23

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We believe our paper poses low risk for misuse, and additionally, we are not releasing new assets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We verify the licenses of our datasets (FFHQ, ImageNet, AMASS) and pretrained models used, and appropriately cite them in the paper in Section 5.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
24

---

Answer:
[NA]
Justification:
We do not release any new assets along with this paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codd/moodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not crowdsource research with human subjects beyond existing datasets (e.g. AMASS).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "yrZj3laHbb", "reviewer": "Reviewer_wQPN", "summary": "This paper tackles the task of sampling from diffusion models with additional inference-time constraints. In this setting, synthesis needs to simultaneously follow the diffusion model-defined generative prior as well as a constraint objective. The paper proposes two techniques to achieve this in a robust fashion. First, trust schedules define the number of optimization steps with respect to the constraint that are carried out between each diffusion model generative denoising step. Moreover, state manifold boundaries prevent the model from falling off the diffusion model-defined data manifold during optimization. These techniques enable more robust and stable constraint-guided sampling of diffusion models. The proposed methods are validated on constrained image modeling tasks (superresolution, deblurring, inpainting) as well as several human motion synthesis benchmarks, and favourable results compared to baselines are achieved.", "questions": "I have one question: In line 173, the authors write that they replace max with ReLU to have a smooth gradient. What is meant by this? ReLUs do not have a smooth gradient, but it is a step function.\n\nI do not have any additional questions, but here are a few typos:\n\n- line 30: constrained -> constraints\n- line 44: it should be \"...state manifold of the diffusion model...\"\n- line 76: I believe the score function misses a $\\log$.\n- line 182: I believe it should be \"LGD-MC\".\n- line 251: \"first first\"\n\nOne additional comment: In line 65, DDPM and DDIM sampling are described as forms of gradient-based MCMC sampling. I do not think this is generally quite correct. For instance, if one choses a deterministic DDIM setup without noise injection, then this is a regular probability flow without stochasticity, but not MCMC. MCMC typically always has stochasticity with some form of noise injection. Broadly speaking, one can see DDPM and DDIM always as a flow plus a variable MCMC component, but not always as pure MCMC. I would suggest the authors to rephrase this statement.", "limitations": "The paper does not have a detailed discussion of its limitations or its societal impact. This did not significantly influence my paper rating, but I would strongly suggest the authors to add a more detailed discussion on both of these aspects to the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Clarity:** The paper is overall written well, clear to read, and easy to follow.\n\n**Quality:** Overall, the paper is of good quality. The paper is clear, has a detailed discussion of related work, as well as mostly appropriate experiments and ablation studies. I do not see any major technical flaws.\n\n**Originality:** The specific proposed techniques are new and original, to the best of my knowledge. They do represent simple heuristics, though, and I have some concerns discussed below.\n\n**Significance:** I think the paper tackles an important task, generation with inference-time constraints, and it achieves strong performance compared to the baselines. This makes the work generally significant.", "weaknesses": "I think there are two main weaknesses:\n\nFirst, both the trust schedules and also the approach to estimate state manifold boundaries are merely well-motivated heuristics and they both require hyperparameter tuning (how many steps in the trust schedule, $\\epsilon_{max}$). For the trust schedules, there is an elaborate derivation that relates approximation errors to state variances (Eq. (9)). But in the end, none of this is tractable and it just motivates the use of a step-dependent optimization schedule, which needs to be chosen fully heuristically. And this schedule also depends on the optimization step size, which needs to be chosen manually, too. The state manifold boundary constraint follows a similar $\\epsilon_{max}$ heuristic. In the end, the paper proposes some useful and well-motivated optimization heuristics, but not an overly novel and rigorous framework for constrained diffusion model sampling. This makes the paper somewhat less original.\n\nSecond, as the authors pointed out in their related work section, there are many previous papers in the area. However, the authors only compare to DPS as well as DPS+DSG, although there should be further applicable baselines. Most importantly, the authors argue that LGD-MC has no code available. However, I believe reimplementing the LGD-MC method should be quite easy, and a comparison to this work would be very appropriate here. Moreover, maybe LGD-MC could even be combined with, or enhanced by the techniques proposed here. This possibility is not discussed.\n\n**Conclusion:** In conclusion, even though I see some weaknesses and evaluation could be expanded, the proposed heuristics seem useful and the paper is of overall satisfactory quality. Hence, despite my concerns, I am carefully leaning towards suggesting acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the clarity, technical quality, novelty, and significance of our work, and raising valuable questions for discussion. Here are our responses to the questions:\n1. Re: heuristics in the method: our paper is heavily inspired by previous works DPS, DPS+DSG and LGD-MC. The focus of these methods, as well as ours, is to design a simple practical algorithm that can be seamlessly integrated into existing Diffusion inference schemes, such that they are *domain-agnostic* as an unified solution for many different domains that utilize the power of Diffusion. With this goal in mind, all these methods strive to distill the insights from theoretical analysis into simple implementations that are computationally tractable and fit nicely into the Diffusion framework. We believe that our results that surpass the current state-of-the-art highlight the practical value of our method. \n2. Re: comparison with LGD-MC: During the rebuttal period, we implemented LGD-MC with results summarized in Tables 1-3 of our one page PDF, where we find LGD-MC performing on par with DPS and DPS+DSG, outperforming other reported baselines on some tasks while falling behind on others. Trust sampling surpasses LGD-MC in almost all metrics across all tasks, highlighting the strength of our method. Please note that while we try to clearly state all parameters required for our method in the writing of our paper, other algorithms may have additional heuristic parameters that are only clear when the implementations are made publicly available.\n3. Re: ReLU: thanks for pointing this out. We apologize for this careless error in writing - $\\max(0, x)$ just means ReLU here. We will remove the “In practice…” sentence at L173.\n4. Re: clarity: We will do a careful pass and also explain DDPM, DDIM, and the limitations of our algorithm better.\nPlease let us know if this addresses your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d76wZJOKo2", "reviewer": "Reviewer_rJ33", "summary": "This paper proposes a trust sampling scheme which incorporates given constraints loss function as guidance for constrained generation. This approach conducts early stop while detecting a mismatch between the predicted noises magnitude of sample and the noisy level at the current state manifold at each diffusion step. In experiment section, this approached is applied to image generation and human motion tasks both across various applications.", "questions": "- In the algorithm, the early termination happens when predicted noise level is higher than the threshold. Shouldn't this be checked at the end of previous step and not applying the gradient from there since the current $\\mathbf{x}_{t-1}^*$ already falls off the manifold? Kindly correct me if I am wrong. \n- I am not sure if image generation tasks are suitable for this paper. I do not see the constraints but simply comparisons with other baselines for checking the performance of the algorithm.\n-  Maybe an ablation study is required here to see how the performance is affected by $\\epsilon_{\\max}$ instead of choosing $\\epsilon_{\\max}$ from the samples generated by base diffusion models since it determines when early stop should be.\n- Could be more typos but not limited to: from Line 172, ``However, we need to make an to handle inequality constraints``, which affects the reading.", "limitations": "Based on the checklist guidelines, the authors claimed the positive and negative societal impact are discussed, but I only found the ``justification`` below the question in checklist is the only place they talked about the impact and I assume this should be mentioned somewhere in the main text or appendix. Limitation is also claimed being discussed in the section 6, and I assume that's mentioned in the second paragraph.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This approach can deal with given constraints using the pre-trained unconditional diffusion models without more training.\n2. Paper is easy to follow.", "weaknesses": "1. It seems that there is no guarantee that the final samples satisfy the constraints since the reverse process will go to the next diffusion noisy level from the early stop if the predicted noise is higher than the threshold while the loss is not 0 yet, which is also reflected in the human motion experiment.\n2. I get the idea that early stop is determined by the norm of predicted noise from the section 3.2: if the norm is large, then $\\mathbf{x_t}$ does not reside in its supposed manifold. But is the reverse direction valid: if the norm is small than some threshold, than $\\mathbf{x}_t$ is on the manifold? Is this necessarily true?\n3. To summarize above, I found the explanation from methodology section tends to give intuitions about why this algorithm would work instead of guarantees or theoretical analysis, such as a bound on the distance between the $\\mathbf{x}_t$ before and after applying loss gradients multiple times, or an upper limit of the probability of samples being out of constraints, etc..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the significance of our work, noting that our paper is easy to follow, and raising valuable questions for discussion. Here are our responses to your questions:\n1. Re: guarantee of constraint satisfaction: the reviewer is correct that our method cannot guarantee precise constraint satisfaction. Our method, as well as current SoTA methods such as DPS and DSG, does not claim such a guarantee, because of the general intractability of $p(y|x_t)$ for intermediate Diffusion states, and general non-linear, non-convex constraints this line of work is dealing with. Even without guarantees, the capability of adding constraints provides a training-free technique that allow us to *control* the generation of Diffusion models in many different ways, all using one unified algorithm. In such cases we believe the value of solid empirical experiments cannot be understated, which is why we went the extra mile in implementing multiple quantitative (and qualitative) benchmarks in two drastically different domains and perform careful evaluation in all these tasks, which is unseen in previous SoTA works.\n2. Re: should the predicted norm threshold be bi-lateral rather than unilateral? While everything is probabilistic here meaning we cannot guarantee $x’$ to be on manifold even if its norm is small, an $n$-dimensional vector with small norm is much more likely a sample from a zero-mean Gaussian than one with large norm. That’s why our threshold is unilateral and only filters large norm epsilons.\n3. Re: checking threshold before iteration: It’s possible that the added noise $\\sigma_t \\epsilon_t$ from the previous step results in the predicted noise level within $x_{t-1}$ to be greater than the threshold, thus falling off the manifold. However, in most cases, the falling off of the manifold is not severe, and the current step’s diffusion step will correct for any deviation from the manifold. This may happen over several timesteps $t$ if necessary, and no loss-guided gradient steps will be taken during these timesteps.\n4. Re: image generation tasks: We will add clarification on this. Such tasks are standard in existing SoTA papers, such as DPS, DSG, LGD-MC, which we borrowed from them. We will make it clear in the main text or Appendix what the constraints exactly are for each task.\n5. Re: ablation: We’ve added a new ablation study in our one page PDF (Table 4) to illustrate why we choose $\\epsilon_\\mathrm{max}$ from samples generated by the base diffusion model. The table illustrates that there is a range of effectiveness for $\\epsilon_\\mathrm{max}$ (roughly between 440-442 for the FFHQ super resolution task), but beyond this range the results do not change too much. Therefore, it suffices to find an $\\epsilon_\\mathrm{max}$ value within the acceptable range, and a general way of finding this acceptable range is by sampling the base diffusion model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dehSJkgBTS", "reviewer": "Reviewer_bnFv", "summary": "The paper presents a method to enhance training-free loss-guided diffusion sampling. The key contributions are:\n\n1. Introduction of Trust Sampling: A novel method called Trust Sampling is proposed, which diverges from the traditional approach of alternating between diffusion steps and loss-guided gradient steps. Instead, it treats each timestep as an independent optimization problem, allowing multiple gradient steps on a proxy constraint function at each diffusion step.\n\n2. Early Termination and State Manifold Estimation: The method includes a mechanism to estimate the state manifold of the diffusion model, enabling early termination if the sample starts to deviate significantly from the expected state. This ensures the proxy constraint function remains trustworthy.\n\n3. Optimization Perspective: The paper reformulates training-free guided diffusion as a constrained optimization problem, providing more flexibility and robustness across various domains and tasks. This approach addresses the limitations of previous methods, such as sensitivity to initialization and performance degradation with fewer neural function evaluations.\n\n4. Experimental Validation: The efficacy of Trust Sampling is demonstrated through extensive experiments in different domains, including image and 3D motion generation. The method shows significant improvements in generation quality and constraint satisfaction compared to existing techniques.\n\n5. Generalization Across Tasks: The paper demonstrates the generality of Trust Sampling across various image tasks (e.g., super-resolution, inpainting, gaussian deblurring) and motion tasks (e.g., trajectory tracking, obstacle avoidance), highlighting its versatility and effectiveness.", "questions": "Please see weaknesses.", "limitations": "The authors have adequately described the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes Trust Sampling, a method that diverges from traditional guided diffusion approaches. Instead of alternating between diffusion and gradient steps, Trust Sampling treats each timestep as an independent optimization problem. This innovation provides a new perspective on training-free guided diffusion, addressing some of the key limitations of existing methods.\n2. The quality of the paper is reflected in the thoroughness of its methodology and the robustness of its experimental validation.\n3. The paper is well-written and clearly structured, making it accessible to both experts and those new to the field.\n4. The significance of the paper lies in its potential to impact a wide range of applications in generative modeling.", "weaknesses": "1. The Trust Sampling algorithm, which takes multiple gradient steps of constraint guidance on each predicted $x_{t}$, is similar to the  corrector stage of PC sampling [1]. \n2. Training-free loss-guided diffusion sampling have been applied not only on inverse problem but also on variable tasks [2-6], such as refined text-to-image and layout-to-image. The authors need to further verify the effectiveness of Trust Sampling on these tasks. \n\n[1] Score-based Generative Modeling through Stochastic Differential Equations\n\n[2] Counting Guidance for High Fidelity Text-to-Image Synthesis\n\n[3] Fine-grained Text-to-Image Synthesis with Semantic Refinement\n\n[4] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models\n\n[5] BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion\n\n[6] LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty of our method, technical quality, structure of our paper, and significance of our work, and raising valuable questions for discussion. \n\nRe: more evaluation tasks: the reviewer suggested several good tasks to further test our method: for example [2] and [4] use neural-network predicted values instead of analytical constraint functions to compute the guidance loss during Diffusion, which can fit into our framework. We will strive to include additional tasks in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Ac9aAQZ5Z", "reviewer": "Reviewer_Pivm", "summary": "This paper addresses this limitation by rethinking diffusion without training loss guidance from an optimization perspective. They formulate a series of constrained optimizations throughout the inference process of the diffusion model. In each optimization, they allow the sample to take multiple steps along the gradient of the surrogate constraint function. The termination conditions come from two aspects: one is the accuracy of the approximate surrogate, and the other is the estimation of the manifold.", "questions": "see Weaknesses.", "limitations": "The authors discuss the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation for this paper seems reasonable to me, and they believe that the one-step guidance of DPS is suboptimal.\n\n2. The proposed direction has improved the performance compared to the two baselines DPS and DSG.", "weaknesses": "1. First of all, I have doubts about the theory of this article. Equation 12 in the article seems to be written incorrectly. As far as I know, $\\epsilon_\\theta(\\mathbf{x}',t)=\\int(\\mathbf{x}'-\\sqrt{\\alpha_t}\\mathbf{x_0})/(\\sqrt{1-\\alpha_t})p(\\mathbf{x}_0|\\mathbf{x}')\\mathrm{d}\\mathbf{x}_0$, it should be conditional expectation here.\n\n2. The motivation for this part is also not very clear. Why does the above integral correspond to a multivariate Gaussian (Line 156)?\n\n3. The number of test samples used in the image experiment is a bit small. Why didn't the author use a data set similar to DPS?\n\n4. In addition, the proposed method still seems to require more than 500 NFE, similar to DPS, and I am a little worried about its efficiency and the need for improvement.\n\n5. DPS itself requires careful tuning of hyperparameters, such as the guidance rate. The proposed method introduces new hyperparameters $m·t+c$ and $\\epsilon_max$, which raises scalability concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the performance of our method and raising valuable questions for discussion. Here are our responses to your questions:\n1. Re: conditional probability in Eq. (12). The reviewer is correct that equation 12 should indeed take the form of $p(x_0|x’)$ instead of $p(x_0)$. We apologize for this careless error and this should not affect the following reasoning.\n2. Re: explain L156-157 better: We see where the confusion comes from and we will revise the text to make it clearer. We are saying part of the *integrand* should be a sample from unit Gaussian, per definition of the forward noising process of Diffusion, not the *integral*. Specifically, we are saying that if we assume $\\mathbf x_0$ to be the true original sample drawn from the data distribution, the portion of the integrand $(\\mathbf x' - \\sqrt{\\alpha_t} \\mathbf x_0)/\\sqrt{1-\\alpha_t}$ provides us the original noise $\\boldsymbol{\\epsilon}$ that was drawn from a multivariate normal distribution $N(\\mathbf 0, \\mathbf I)$. However, since we're not sure which $\\mathbf x_0$ is the correct one, we perform a \"weighted sum\" (an integral) of $\\boldsymbol{\\epsilon}$ over all possible $\\mathbf x_0$ starting points, in which each term $\\boldsymbol{\\epsilon}$ is a sample of a normal distribution. Of course, these samples from normal distributions are not independent, but because each term is sampled from a normal distribution, each individual term is unlikely to have a high norm. Therefore, for the overall integral to have a high norm, the individual terms must be aligned along a particular direction. This indicates that a high $||\\boldsymbol{\\epsilon}_\\theta ||$ means an out of distribution $\\mathbf x'$.\n3. Re: same test set size as DPS: Unfortunately, reproducing DPS with the same test set size of 1,000 images would take around 20 hours for FFHQ and 100 hours for ImageNet for each configuration of parameters. Note that our paper includes many more quantitative ablation studies than the DPS paper. That said, we will update all numbers in our tables with the DPS test set size in our final version. This update should not affect the overall narrative due to the significant quality gap between ours and the baselines.\n4. Re: efficiency: We agree there can be further room for improvement in terms of efficiency, though the focus of our evaluation is to showcase that with a similar level of NFEs, our method gives higher-quality results (as shown by quantitative metrics and qualitative comparisons) than existing SoTA methods.\n5. Re: the introduction of two additional parameters. While our method has additional parameters, the robustness of our parameter space is very different. DPS indicates high sensitivity to the hyperparameters such as guidance rate (also pointed out by the LGD-MC paper), and DPS+DSG is very sensitive to several hyperparameters such as their guidance rate and interval. On the other hand, trust sampling isn’t nearly as sensitive: Tables 4 and 5 indicate a wide range of parameter choices for our trust schedules, most of which perform better than baselines. Furthermore, our new table (Table 4 of our one page PDF) shows $\\epsilon_\\mathrm{max}$ has a wide range of applicability as well. Therefore, despite there being more parameters, our method is rather robust with respect to our parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive feedback. We are encouraged by the reviewers’ recognition that our proposed method addresses the important task of Diffusion generation with inference-time constraints, which is widely applicable across many domains. Additionally, our experimental validation spans a diverse range of tasks not seen in previous papers (2D images and 3D human motion), and demonstrates robust performance improvements over recent baselines such as DPS and DPS+DSG. In terms of methodology, reviewers generally agree that our proposed algorithm, which frames guidance as multi-step gradient-based optimization, is both novel and theoretically intuitive.

Reviewers also made a few great suggestions on additional experiments to strengthen the evaluation of this work. We did our best given the tight rebuttal timeframe to implement the following new experiments: 
1. We added LGD-MC $n=10$ and $n=100$ baseline results to Table 1 (FFHQ), Table 2 (ImageNet), and Table 3 (motion task table). Trust sampling outperforms LGD-MC on almost all tasks.
2. We performed a new ablation on $\epsilon_\mathrm{max}$, illustrating the robustness of our parameter space. 

See our one page PDF for these updated tables. We provide detailed answers to specific questions in the individual responses.

Meta-review: All reviewers are positive  on the paper after rebuttal.
They find the paper well motivated and the method original albeit simple.
The rebuttal addressed most concerns
Decision: Accept (poster)