Fast constrained sampling in pre-trained diffusion models
Authors: Alexandros Graikos
Nebojsa Jojic
Dimitris Samaras
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion models
training-free conditional sampling
Primary area: deep_learning
TL;DR: We improve and speed up training-free conditional sampling from diffusion models using an approximation to the Newton method

Abstract: Large denoising diffusion models, such as Stable Diffusion, have been trained on billions of image-caption pairs to perform text-conditioned image generation. As a byproduct of this training, these models have acquired general knowledge about image statistics, which can be useful for other inference tasks. However, when confronted with sampling an image under new constraints, e.g. generating the
missing parts of an image, using large pre-trained text-to-image diffusion models is inefficient and often unreliable. Previous approaches either utilized backpropagation through the denoiser network, making them significantly slower and more memory-demanding than simple text-to-image generation, or only enforced the constraint locally, failing to capture critical long-range correlations in the sampled
image. In this work, we propose an algorithm that enables fast, high-quality generation under arbitrary constraints. We show that in denoising diffusion models, we can employ an approximation to Newton’s optimization method that allows us to speed up inference and avoid the expensive backpropagation operations. Our approach produces results that rival or surpass the state-of-the-art training-free
inference methods while requiring a fraction of the time. We demonstrate the effectiveness of our algorithm under both linear (inpainting, super-resolution) and non-linear (style-guided generation) constraints. An implementation is provided at https://github.com/cvlab-stonybrook/fast-constrained-sampling.

Fast constrained sampling in pre-trained diffusion models
Alexandros Graikos Stony Brook University, Stony Brook, NY
Nebojsa Jojic Microsoft Research, Redmond, WA joj wwgiimicssooot.ccmmm com
ggraikos@cs.sstnyybroo edu
Dimitris Samaras Stony Brook University, Stony Brook, NY
samaaras@cs.stooybroooo edu
Abstract
Large denoising diffusion models, such as Stable Diffusion, have been trained on billions of image-caption pairs to perform text-conditioned image generation. As byproduct this training, these models have acquired general knowledge about image statistics, which can be useful for other inference tasks. However, when confronted with sampling an image under new constraints, e.g. generating the missing parts of an image, using large pre-trained text-to-image diffusion models is inefficient and often unreliable. Previous approaches either utilized backpropa- gation through the denoiser network, making them significantly slower and more nemorr-demanding than simple text-to-image generation, or only enforced the constraint locally, failing to capture critical long-range correlations the sampled image. In this work, we propose an algorithm that enables fast, high-quality gen- eration under arbitrary constraints. We show that denoising diffusion models, we can employ an approximation to Newton's 's optimization method that allows us speed up inference and avoid the expensive backpropagation operations. Our approach produces results that rival or surpass the state-of- ofht-----trerr training-fiee free inference methods while requiring fraction of the time. We demonstrate the effectiveness of our algorithm under both linear (inpainting, super-resolution and non-linear (style-guide generation) constraints. An implementation provided this GitHub repository
Introduction
The development of large text-to-image models 22 27 26 30 has made denoising diffusion [32 [6
the go-to approach for capturing complex data distributions in high-dimensional spaces, such as images. By training on billions of text-image pairs, these models have acquired general knowledge about the image space, beyond text-to-image generation. This knowledge is useful in quickly adapting
new conditions 455 40 and utilizing model features to solve downstream image tasks [38 39).
The simplest way to utilize this prior knowledge is by fine-tuning the model. However, fine-tuning may require non-trivial computational resources and thus, previous works have focused on developing algorithms for conditional generation using only the pre-trained model 5
28 6
[3]. These
methods modify the diffusion sampling process by computing additional gradient terms that move the sample towards the condition while denoising. When these gradients are computed using backpropagation through the model weights, there is significant increase in inference time. On
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Constraint
PSLD 8min 28
FreeDoMM 1min
MPGD 18s
Ours 15s
Ground Truth
44
13
When tasked with completing the missing half of an image, previous methods are slow and fail to capture the important long-range dependencies between pixels. The proposed algorithm generates a reasonable image at a fraction of the time.
Figure I:
the other hand, when attempting save computation by not propagating the condition information through the model, the generated image fails to capture the necessary long-range correlations.
As example, in Figure Methods that backpropagate through the denoiser model (LDPS, PSLD [28], FreeDoM [44]) require significantly more time to run and do not consistently produce realistic results. Algorithms that do not compute gradients through the denoiser (MPGD (3] fail to propagate the condition to distant pixels. With the shortcomings of existing approaches in mind, we pose the following question: Can we avoid backpropagation through the model weights while at the same time maintaining long-range consistency when updating the sample with the conditioning signal?
different algorithms fill in the missing half of image.
We view the denoiser network as a function that removes all noise from an input image. In that context, the goal of constrained sampling is to find how the input should change such that the clean output satisfies the condition better. This involves the denoising function's 's Jacobian matrix, which transforms a local change in the input to a change in outputs. We show that existing methods that backpropagate the constraint through the model employ the transpose of the Jacobian to invert this transformation, corresponding the gradient descent direction.
alternative direction given from the Newton method (24, but requires more computation as it involves finding the Jacobian inverse. However, the input and output of the denoiser are closely related; when the intensity some pixels in the final, noise-free image is increased, the same pixels are also expected to brighten in the noisy image and vice versa. Therefore, we propose to approximate the Jacobian inverse the Newton step with the Jacobian matrix itself. We show that this is valid approximation that (a) is fundamentally different from the gradient descent direction used in previous works, (b) is cheap to compute, with only two forward passes through the denoiser required, and (c) can quickly converge to the desired solution in large pre-trained diffusion models.
To evaluate, we generate images under both linear and non-linear constraints. We first show that our approach matches the results of state-of-the-ar methods on free-form inpainting and 8x super- resolution at a fraction of the inference time. We then demonstrate how existing methods fail at inpainting large regions, while our algorithm obtains results closer to fully fine-tuned diffusion model on inpainting. Finally, we show how we can apply our algorithm to non-linear constraints and perform style-guided and mask-guided generation, where the proposed method consistently generates images that satisfy the constraints better than existing approaches.
2 Background
2.1 Denoising Diffuuson Models
Denoising diffusion models 332 16 have been widely adopted due to their exceptional ability to synthesize diverse and high-quality samples. The original formulation treats the training and inference process hierarchical latent variable model æT TT-1 - latent is distributed normally ar ~ N(0 I) and p(ar)) represents the data distribution. Given noise schedule a; that defines the forward transitions æ, Φ++1 that corrupt the data with Gaussian noise, usually centered at fou xt and with variance (1 د ), the model is trained to reverse each step the diffusion process by predicting the noise added to the clean sample. The predicted noise is shown to approximate the score function Va, log pr (xx) (34] of the diffusion latent variables.
æ1 Zg. where the final

---

Further iterations of denoising diffusion introduced classifier guidance [9) which adapted a pre- trained unconditional diffusion model for conditional sampling by training an additional classifier p(y (axa) During inference, each reverse step also includes the gradient Va, log p(y)z)) which guides the diffusion latent æt towards regions that also satisfy the condition y- However, if the conditioning y is already given, training this additional classifier on top of the diffusion model is costly. Classifier- free guidance [15] eliminated the need for an additional classifier by incorporating the conditional guidance into the base diffusion model training process.
The most widely adopted formulation of denoising diffusion is Latent diffusion models (LDMs) 2 LDMs reduce the complexity by modeling the compressed latent space of an autoencoder instead of images. With an encoder & and decoder D that accurately reconstruct images zo (E (xx))) the diffusion process can be made more efficient as redundant information in an image is left for the decoder to reconstruct. Most large text-to-image diffusion models, such as Stable Diffusion 27 are based on the LDM approach.
2.2 Gradient descent steps for constrained sampling
Previous works studied whether large pre-trained diffusion models, which required significant investment to train, can be directly used for inference under novel conditions without additional tuning for each different constraint [5]. The typical problem formulation is denoising the sequence ær, -1-1, 11,
21, Ey under a linear constraint on the final signal in the form Axg y. or relaxed version that minimizes || Aro - U112 as part of the likelihood p(y| = (w; Azp,o"1)
Contrary classifier guidance, which trained separate model for the likelihood p(y axx), the linear constraint only applies to the final, noise-free image Vo. Thus, existing methods rely on Tweedie's formula [TO), by which denoising diffusion models approximating Vz, log (ar,) can be used to express the expected value of zy. denoted as Zo. Including the constraint as if an additional observed variable y was generated requires adding Vz, log p(y xx) to every diffusion sampling step, and
previous works considered different approximations of p(y xt) using the estimated to [4)
Regardless of how the constraint gradient is applied, the regular denoising diffusion steps are altered so that at each f, the generated latent x; is moved in the direction reducing the cost
C(x.)= (Aio(æ:)) - (A -y)
(1)
For example, in the case of inpainting missing pixels, the matrix A extracts the subsection of the known pixels in image ão to be compared with a given target The estimated expected value of xo the end of the chain is provided by the diffusion model as a nonlinear function (o(x,) learned by the denoiser network during training. Typically, these moves are gradient descent moves, i.c. moves of æ in the direction of ee where
e = V C(x,) JT AT (Aoo - y) = JT J = Vn, #o(())), e = AAA - y)
Note that we name the error signal e as the (negative) direction of the gradient w.n.t. xo itself, and the e, is the matching move in the noisy Tt.
The matrix AT [ inverses the operation of A and can usually be computed for each task a-priori. Since computing the full Jacobian J is impractical, the gradient is instead computed using backpropagation through C(xx). which yields the direction - JT e. Chung et al. [5. [6. for example, apply gradient steps of this form to x; at each step t of generation before moving on to the next stage. As optimization of æ. might reduce the total noise in the image below what the denoising at t- - was trained for, the gradient steps moving &; towards optimizing C(ax,) can be combined with adding additional noise, which could also be seen as a form of stochastic averaging, as done by Yu et al.
3 Method: Approximate Newton steps
We start by observing that Newton optimization step, instead of moving H, in the gradient descent direction J e, moves it in the direction J J-'e. We demonstrate it on a more general form of the cost
C(x,)) (f(to(=;)) - - (f(oo(zz)) 4).
(3)
3

---

for some target y to be matched with projection function f. which in the linear case is f(x) = Ax.
The Newton optimization would first approximate
f et)) = (z)) J Jer.
(4)
where y is the Jacobian of f. We then rewrite the cost of move -e as
Cl et) (f( J,Je, v) " (f(00(zz) - Jj yy)
(5)
The cost is minimized by setting Ve, C = 0 to get the system
J J" (s(ooZz)) y) = J J JjJer - -- -g) y)
(6)
e, e- e j ()( - y)
(7)
assuming the inverses exist. In the case of f(x) Ax for inpainting and super-resolution tasks, where y is lower-dimensional, the inverse of J,. not defined, but we can use the pseudo-inverse. Similarly, for a non-linear f. we can compute using numerical methods, e.g. backpropagation through f when it a neural network, which would approximate e using J..
Therefore, is the same in gradient descent and Newton optimization, but its relationship e, differs
(a) GD e, J (b) Newton
Je,=c
(8)
Gradient descent can be computed without directly evaluating the Jacobian by backpropagation on the scalar cost C. For the computation of J-le, on the other hand, we have no such method.
In cases where computing the inverse of the Jacobian is prohibitive, inexact Newton methods 7 propose first finding an approximation e, to the solution of Eq (8) (b). performing the Newton step using the approximate solution, and reiterating until convergence. When the residual r is strictly reduced every step, inexact Newton methods converge the correct solution
|rllz
e Je
< k,, E [0, 1).
|el|a
Since we know that the input and output of the denoiser are related by additive Gaussian noise, we propose the inexact move
e, = Je
(10)
where instead of computing the Jacobian inverse, we use the Jacobian-error vector product. If the Jacobian tells us how to transform a local change in the input to a change in the output, then we posit that, for denoising diffusion models, we can use the same transformation for the inverse. Substituting ef with the proposed update, adding a learning rate À to control convergence, we get the residual
r=c XJ = (I - XJ
(11)
For this residual to be strictly reduced at every iteration of our algorithm, we require
| ll |I |ell|
Jel)ell
||
XJ l2lleel = I AJ < <1
(12)
||e|l2
||l||
As long as the spectrum of the denoiser Jacobian matrix is bounded, we can choose small enough that strictly reduces the residual. However, the proposed move would converge slowly if the only allowed were very small. In practice, we find that we can use large learning rates, making the convergence similar or even better than gradient descent. In Appendix A.] we present analysis of the spectral properties of the Jacobian matrix of the Stable Diffusion denoiser used in our experiments. Furthermore, in Appendix A.3 we demonstrate the differences between our proposed inexact and the exact Newton method on a small-scale experiment.
The direction of optimization we propose in Eq. (TO) has another advantage over the gradient descent update. The direction e, can be computed numerically to save both on computation and memory compared to using backpropagation on the cost in Eq. (T. To derive the update, consider the function h(s)) L0 + se) where the variable s is scalar. derivative s = is
dh
Je,
(13)
ds
==0

---

y
S00
1=600
We showcase the difference between the proposed method to compute the update direction (J) and gradient descent (J²). The heatmaps indicate where the input at would change when perturbing a single pixel in the output, denoted in green. The two directions are considerably different, with ours capturing better longer-range correlations and maintaining shapes. Even though we use finite differences, the direction computed from J is sharper in some regions, like the outlines.
Figure 2:
and so the direction e, can be computed using the numerical derivative of h(s) at =
dh
h(6) - h(0) Ebl Lp a + ge) Zω (xz)
= AJe
(14)
ds
5
1==0
requiring no backpropagation but instead two forward passes through the network: one to compute (2;;) and another to compute do for the H, perturbed in the direction of the error vector.
is important to point out that the theoretically optimal (x())) would have symmetric Jacobian, as 2o approximates the true expectation E [[xo|zze this is indeed symmetric
Vay [ + Uz V.Va vx log Pr (2x)]. and the gradient of
(15) This would render the gradient descent and the proposed update equivalent. Yet, we find that the trained denoiser models do not satisfy this ideal condition, making the two update directions noticeably different.
x. D x0
I vs V log)r(x;))) Var
In Figure 2. we visualize this difference between J and J by computing rows of J and J and averaging the magnitude of all matrix values that contribute to each pixel. This indicates that there is a difference in which pixels in 2, would change when perturbing a single pixel in εu(œ;) for the two update directions. Further experiments on the difference between the two update directions are included in Appendix A.2 Whether the denoising diffusion models are trained with score matching in mind 34 or using the variational method of Ho al. [16), they do not directly optimize to match the real score V., log Pt (xx) everywhere nor are they constrained to produce symmetric Jacobians. We hypothesize that this discrepancy in the two update directions is another reason why our optimization of may be more suitable for some applications, which we demonstrate in the experiments (Section 4b.
Lastly, the proposed update of Eq. (TOp only requires us to provide the direction of the error e, which points locally towards a lower cost for the constraint on Zy. For linear constraints such as inpainting, this direction can be obtained in closed form using the inverse of the operator expressed by A. For non-linear constraints, does not have to be computed in closed form from J, and can be approximated. In our experiments, we use backpropagation through the differentiable non-linear VAE decoder and constraint, which is cheaper than backpropagating through the denoiser.
An alternative would be to utilize another inexact Newton approximation to avoid backpropagating through the constraint altogether. We discuss this in Appendix A.4 where for linear constraints applied to the the VAE output space (i.e. pixels), we propose an inexact Newton method that utilizes the Jacobian of the VAE encoder to approximate the inverse of the VAE decoder Jacobian. Similar
5

---

Algorithm The proposed algorithm for sampling under linear and non-linear constraints.
1: Input: Pre-trained diffusion model &o (æt), linear constraint C y) = [ - y or non-linear constraint C( y) = (an (xx)) vlÉ. condition y, step size ö, iterations K, learning rate X, diffusion step size s and schedule parameters ag B,
2: 2T ((. I)
3: for = T, s,T ........ do
4: for 1,2,..., K do
5: Linear then A" A'"AAAaiii (Aig((r;) y)
6:
(Using pseudoinverse A²)
7: else if Non-linear then
8:
- J (J(+)(2))) - 9) do C( y)
[Using backpropagation through f)
9:
end if
10: e, [ + òe) - /6
11:
Dt Dit det
12:
end for
13:
21 N(O. I)
var-s Fo((::)
14:
6t VI-ai-a Dtt V I--s--n
15: Dt-s / / t + V1 04-s - BI-oes ct + Be-s2z
(DDIM step)
16: end for
17: Return: 250
approximations have been discussed in the literature before 36 and can be particularly useful when backpropagation is infeasible [4]]
The proposed method for sampling with linear and non-linear constraints is described in Algorithm 1
using DDIM as the diffusion sampling algorithm [3) Using other diffusion sampling algorithms [9]
intuitive by interleaving the diffusion and our constraint gradient updates on æ, (Appendix [A.6]
4
Experiments
4.1 Linear Constraints
We first verify our algorithm by generating images under linear constraints, which has been the main
application of many previous algorithms [8.28, [). We follow the evaluation setting of Saharia et al
(29] and test our method on ImageNet [8]. using the first 1000 images from the 10k validation set of the ctest100 Ok split. For evaluation, measure the PSNR, LPIPS [46), and FID [4 between the
real and generated images. We use Stable Diffusion 1.4, which pre-trained on the LAION [31 text-image pair dataset. Experiments were
NVIDIA RTX A5000 24GB GPU.
Free-form inpainting and 8x super-resolution In free-form inpainting, masks are randomly sampled and mask 10-20% of the image pixels. For inpainting with method, opt to directly operate on the Stable Diffusion latent given that Stable Diffusion VAE mostly compresses information locally. We apply the masking in pixel space, encode the masked image and inpaint with the unmasked VAE latents. We also apply a 3 x 3 dilation kernel on the pixel mask before downsampling, masking out some extra pixels along the edge that we find the VAE fails encode.
For roper-resolution, we cannot apply the constraint in the VAE latent space since image downsam- pling does not correspond to downsampling VAE latents. This makes super-resolution linear. It involves the differentiable decoder D. and to compute the error direction e, we backpropagate the pixel-level linear constraint (AD(#,))
(AAD((( (.)) --y) yy through the decoder network. We discuss backpropagation for computing the error direction further in the non-linear constraint experiments (4.2 but leave super-resolution in the linear section done by previous works.
For inpainting, we set the number of optimization steps K = over which we linearly decrease the learning rate À from 0.5 to 0. For super-resolution, we use K 10 and constant À = 0.1. For both degradations, we also include additive white Gaussian noise with og 0.05, use DDIM 33 steps and normalize the computed gradient e, with its cc-orm.

---

Image
IDPS
PSLD
P2T.
Ours
Ground Truth
Free-fore
Super-resolation
Figure 3: Comparison between method and existing algorithms free-form inpainting and 8x super-resolution. We directly use the images and results from 6 since there is no code available to replicate their method.
Table I:
Quantitative evaluation (PSNR, LPIPS, FID) on free-form inpainting.
Table 2:
Quantitative evaluation (PSNR, LPIPS, FID) on 8x super-resolution.
Inpaint (Free-form)
Super-res (x8)
Method
PSNR
LPIPS 1. FID | Time
Method
PSNR 1 LPIPS 4 FID
Time
P2L+ captions
66
21.99
0.229
32.82
30m
P2L captions
23.38
0.386
51.81
30m
LDPS
21.54
0.332
46.72
6m
LDPS
23.21
0.475
61.09
6m
PSLD
28
20.92
0.251
40.57
8m
PSLD
28
23.17
0,471
60.81
8m
Ours
21.73
0.258
19.39
15s
Ours
24.26
0.455
60.99
1m
Ours* captions
24.95
0.405
44.74
Im
We present the results on free-form inpainting in Table We find that our method generates better- aligned parts for the missing image regions, reflected in the significant improvement FID, At the same time. our algorithm maintains consistency with the given image parts, which is reflected in the similar PSNR and LPIPS scores the baselines.
For 8x super-resolution (Table although the improvements are smaller, we attain similar quality and faithfulness the generated images at fraction of the inference time. The best best-performing baseline, P2L (6), also utilizes a PaLI VLM (3] to caption the low-resolution images before the diffusion inference. We believe that the main advantage of P2L comes from introducing text conditioning, whereas all other methods rely only on the unconditional model. To verify this assumption, we run our algorithm for x8 super-resolution with text prompts generated from the downsampled images using Qwen-2.5 -25 12 as the VLM. The results clearly demonstrate that we can indeed bridge the performance gap to P2L, and even improve upon the PSNR and FID metrics when introducing text captions to the super-resolution process.
Overall, the main advantages in inference time and GPU memory. We achieve similar better results to every other method while only requiring a fraction the compute. When we do not backpropagate through the VAE decoder (inpainting), our method requires only 15 seconds. In super-resolution, where we run more optimization steps and backpropagate, the time increases to report
minute. In comparison, P2L requires 30 minutes. Since P2L has no public implementation, the results directly from the paper and estimate the inference time as 5x that of LDPS [], which the authors mention as a reasonable expectation. Regarding memory, for single image, our forward passes only require ~ 9GB of memory, compared to backpropagation, which consumes ~ 17GB.
Box Inpainting In the previous experiments, we observed that our method performed better in inpainting, which required the model to infer more missing information in the given image than in super-resolution. We further push the model by asking inpaint a box that covers 25% of the input image. In the comparisons, also include FreeDoM 44, which, although not previously shown
7

---

Masked Image
LDPS
PSLD
FreeDoM
Ours
Inpainting Model
Image
Figure 4: Qualitative evaluation of large area (box) inpainting on ImageNet. Our method achieves results closer to the fine-tuned inpainting model while requiring a fraction of the time to run per image compared to baselines.
inpainting, claims to be fast training-frre free inference approach for any condition. We also include the fine-tuned SID-Inpaint model (27), which required 500k additional training steps.
We present the results box inpainting in Table and Figure 4 Our approach outperforms all existing training-free free methods in all metrics and the closest to the fine-tuned SD-Inpaint model, which we consider an upper limit. Qualitatively, lot of denoising steps, generate blurry parts that align with the average appearance over the rest of the image. FreeDoM, the faster baseline, although generating non-blurry parts, frequently fails to maintain consistency with the rest of the image. Our method achieves both high quality and consistency shorter time all previous methods backpropagate through the denoiser weights.
find that LDPS and PSLD, which perform
Number of steps and learning rate We ablate the number of optimization steps K and learning rate hyperparameters of our algorithm on the box inpainting task. The quantitative results are reported in Table where we find that the original choice of K = 5 steps and À = 0.5 achieve the best reconstruction and image quality metrics. In Appendix Figure 14 we present qualitative results where we observe that running fewer optimization steps gives blurrier results, which is expected as known regions of the image also seem to not have converged to the given values. Using a higher learning rate leads to the model sometimes 'overshooting' by inpainting the missing regions with realistic-looking parts that do not necessarily the rest of the image.
Finite difference approximation In Appendix A.5 we ablate the finite difference step size d, and compare to the exact Jacobian- product computation. We find that our method robust to the choice of 8 and yields similar results to using the exact computation, while requiring less time.
4.2 Non-linear Constraints
Our algorithm can be applied to any non-ling non-linear differentiable constraint. The Newton derivation for non-linear constraints in Eq. involves inverting two Jacobian matrices, the denoiser Jacobian J and the Jacobian of the constraint function f. .J. Our algorithm offers a way to approximate J For J we use the gradient descent direction JT. which computed using backpropagation through the network f. Computing the gradient descent direction for f does not require backpropagation through the denoiser, only through f. Thus, for non-linear constraints, combine proposed Newton direction for the denoiser with gradient descent for the differentiable constraint.
In this section, we showcase style-guided generation with our algorithm. In Appendix B. we
present results on mask-guided generation, where we show that our method outperforms the strongest baseline, MPGD [13. We discuss non-differentiable constraints in Appendix B.3

---

Table 3: Quantitative evaluation large area (box) inpainting. Our method outperforms all previous baselines and is the one closest to fine-tuning the diffusion model for inpainting (SD-Inpaint). When using fewer steps K, our algorithm does not suffi- ciently converge. When using very high learning rate X, it overshoots, adding non-realistic parts.
Table 4: Quantitative evaluation of style genera- tion. The style score what the gradient steps are directly optimizing for when using CLIP. Our approach achieves better style scores than the baselines without compromising faithful- ness to the prompt, even when using a different model to guide style (OpenCLIP).
Inpaint (Box)
Method
Style Score CLIP 1 Time
Method
PSNR
LPIPS
FID J Time
DDIM
761.0
31.61
9s
LDPS
17.52
0.42
76.32
6m
FreeDoM
498.08
30.14
80s
PSLD 28
17.30
0.38
74.02
8m
MPGD 13
441.00
26.61
50s
FreeDoM
16.18
0.42
55.68
Im
Ours (w 2)
368.37
23.95
45s
Ours (K
0.5)
18.30
0.30
42.01
15s
Ours (w 5)
310.96
24.57
458
Ours OpENLUPP
(w = 5)
434.45
25.94
45s
Ours (K 2, À
0.5)
18.01
0.39
68.75
7s
Ours (K À 1.0)
17.48
0.32
47.20
15s
SD-Inpaint
19.05
0.28
32.93
4s
Style
Ours
Ours (OpenCLIP)
FreeDoM
MPGD
"a car wearing glasses"
Figure 5: We guide the style of Stable Diffusion images with CLIP (or OpenCLIP) model, using classifier-free guidance w = 5. The images generated by our algorithm are closer to the reference style while maintaining faithfulness the text prompt.
Style-guided generation The goal is generate an image that simultancously follows the style of a reference image Øref and a given text prompt. Following previous works [44] 13] which also perform style-guides generation, reference and the generated images while denoising with a text prompt. We use the 2nd CLIP layer features define the cost C' as the Frobenius norm of the the Gram matrix difference C' (CLLnCCLLP.p....
match the statisties of CLIP 25 features between the
- (eram(CLIP2
Adhering to the evaluation of Yu et al 44 we use 1000 random pairs of reference style images from WikiArt [37] and prompts from PartiPromtps [43]. We measure the CLIP similarity between the generated images and the text prompts to evaluate the faithfulness to the text condition, and the final difference between the Gram matrices to evaluate the faithfulness to the style reference (style score). We use the CLIP ViT-B/16 model for guiding the style of the image and evaluating. We also repeat the experiment using the OpenCLIP VIT-B/32 model 4 for guidance. We perform K = gradient updates for every demoising step, using linearly decreasing learning rate À from 0.5 to 0.1 and classifier-fie free guidance [13] w 2 and w = 5 for the denoiser.
The results are presented in Table and Figure 5 Our method is best at minimizing the constraint, which is the style score evaluation metric. Although in general, the lower the style score the more difficult it is to maintain high CLIP similarity with the prompt, we observe that our method balances

---

well between the style and text, especially when we increase the guidance weight. When using an OpenCLIP model to guide the style, we achieve a better style score than the baselines that optimized directly for it with CLIP This also indicates that we minimize the target cost without generating adversarial artifacts that trick the CLIP evaluation.
Excluding the non-guided DDIM, our method is as fast as MPGD [3]. which does not backpropagate the style loss through the denoiser network, but only modifies the æn estimation at every denoising step. By only adjusting the to prediction, MPGD completely fails to propagate the constraint to distant pixels (Appendix B.7 making it unusable in other constrained sampling settings.
Mask-guided generation We guide the Stable Diffusion model with a separately trained face segmentation network. We employ off-the-shelf model and set the constraint C to be the KL divergence between the per-pixel segmentation classes predicted for a reference image and the generated image. Using 100 images from the FFHQ (17) validation set,
both method and MPGD [13]. which is the fastest baseline that works very well with 'dense" constraints, i.c. constraints that are applied to all pixels.
The results in Table [ show that with a simi- lar compute budget, method achieves both faithfulness (mloU between generated and ref- erence images) and image quality (CLIP-FID). We used CLIP-FID because it performs better than Inception-FID on small set of images 18. MPGD with a high weight (p) 'burns in' the segmentation mask, leading artifacts and non-realistic images, whereas a lower weight does not produce images faithful the mask.
Table 5: Mask-conditioned generation using 100 FFHQ validation set images reference, using Sta- ble Diffusion with the prompt 'a headshot photo'.
Method
mloU t
CLIP-FID J
DDIM
0.09
48.78
MPGD 13 (p = 1)
0.47
77.11
MPGD (p 0.5)
0.36
56.38
Ours
0.42
59.79
We provide qualitative results of our segmentation mask maske generation experiment in Appendix Figure 13 The baseline (MPGD) either over-satisfies the constraint by burning in artifacts (p 1) or fails to generate images that adhere to the constraint (p = 0.5). Our approach generates the most realistic images while also getting the mask prediction match the reference image. Using Stable Diffusion, we chose the text prompt 'a headshol photo' to constrain the generated images. Considering the limitations of generating faces with Stable Diffusion, the results may not be par with an FFHQ-specific model, but we still find our algorithm able to synthesize more usable images than the baseline, even in this difficult case.
Limitations Using high classifier-free guidance (w > 10), which is useful in some training- free tasks such as multi- view inference [42]. requires to significantly reduce the learning rate X. The high guidance alters the spectral properties of the denoiser Jacobian, making the model more sensitive to small changes in the input. Another limitation highlight is with distilled models (35] 21 that perform inference in fewer (1-5) steps. We observed that our algorithm requires more steps to achieve comparable results (Appendix B.4). mitigating the inference speed benefits of distilling the model.
5 Conclusion
We presented new algorithm for inference under arbitrary constraints in pre-trained diffusion models. Our approach exploits the relationship between the noisy image input and clean image output of the denoiser to approximate the Newton optimization steps with cheap forward passes. The images generated under linear and non-linear constraints are comparable to or better than state-of-ther -hee-artt methods, at a fraction of the inference time. We offer a practical algorithm to sample from large pre- trained generative image models under any condition, with the potential to enable new training- free downstream applications that rely on strong image prior.
Acknowledgments
Part of this work was done during an internship at Microsoft Research Redmond. This research was also partially supported by NSF grants IIS-2123920, IIS-2210044
10

---

References
[1] Walter Edwin Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly applied mathematics, 9(1):17-29, 1951.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint wrXi:22502.13923, 2025.
[3] Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov. Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski. et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint 2rXiv:2310.091 2023.
[4] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 2818 8--8,2,, 2023.
[5] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Inter- national Conference Learning Representations, 2023. URLhttps:: httph:ttp/////t /ttpss///ppereeeeeeee .ett/ Idrum? dnnDunDAAGOD
[6] Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8941- 8967. PMLR, 2024.
httppp::///ccceeiiggeee mlr. press/v235/ 2tttss//22//chugg2bb chung24b .html
[7] Ron S Dembo, Stanley C Eisenstat, and Trond Steihaug, Inexact newton methods. SIAM Journal on Numerical analysis, 19(2):400 408, 1982.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances neural information processing systems, 34:8780-8794, 2021.
[10] Bradley Efron. Tweedic's formula and selection bias. Journal of the American Statistical Association, 106(496):1002-1114 2011.
[11] Leon A Gatys, Alexander S Eeker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2414-2423, 2016.
[12] Henri P Gavin. The levenberg-marquard algorithm for nonlinear least squares curve-fitting problems. Department of Civil and Environmental Engineering Duke University August, 1-23, 2019.
[13] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Representations, 2024. URL https://opennreview wtt.rrrwwwwweeeeeeoooorroiooo... wtt:/formmmddd0BBB0UU.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 12X7v:2207.12598 2022.
arXiv preprint
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.

---

[17] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1401-4410, 2019.
[18] Tuomas Kynkäänniemi. Tero Karras, Miik Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. The Eleventh International Conference on Learning Representations, 2023.
[19] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022. URL https://openreeveww weeee//oorumndddPPXWWaappp
[20] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[21] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
[22] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text- text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804, PMLR, 2022.
[23] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high nigh-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
hwt.hhtttp///ec httt:::///peerevieww net. forum? lofoorummddiiiii 52zR8xgf
[24] Boris T Polyak. Newton's 's method and its use in optimization. European Journal of Operational Research, 181(3):1086-100 1096, 2007.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-8763 8763. PMLR, 2021.
[26] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 27X1::220..06125, 1(2):3, 2022.
[27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[28] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL attps://openrevall www.eeeefoonni -itpmmmmddxxXFFFFF- XeBF-XXBFFRdROOOo
[29] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusson models. ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[30] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[31] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 08225278-55294 2022.
12

---

[32] Jascha Sohl- Dill-Disstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[33] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2020.
[34] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL luttps://pperrevai
[35] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 32211 1-32252. PMLR, 2023.
[36] Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, Lea Zimmermann, and Ullrich Köthe. Lifting architectural constraints of injective flows. In ICLR, 2024.
[37] Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, and Kiyoshi Tanaka. Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 28(1):394-40)) 2019. doi: 10.1 2TT.TIPP.018.22666666 URL https: /tt/dii.org///// 10. 1109/ TIP. 2018. 2866698
[38] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363 1389, 2023.
[39] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3554-3563, 2024.
[40] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint (eXiv2230800672.1 2023.
[41] Srikar Yellapragada, Alexandros Graikos, Kostas Triaridis, Prateek Prasanna, Rajarsi Gupta, Joel Saltz, and Dimitris Samaras. Zoomldm: Latent diffusion model for multi-scale image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23453 23463, 2025.
[42] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7796-6807) 2024.
[43] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku. Yinfei Yang. Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
[44] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffuuson model. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23174-23184, 2023.
[45] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction we clearly state our contribution of new algorithm for constrained sampling pre-trained diffusion models, and how improves over existing baselines on set of linear and non-linea tasks. We also refer to the mechanism of our algorithm, approximating the Newton steps, which is shown in the method section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We included a paragraph describing the limitations of our method concerning its applicability tasks that require high classifier-free guidance and on models distilled from the base, large diffusion model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our main contribution is showing how we can approximate the Newton steps with a cheap and fast alternative. We show why this approximation is principled using the inexact Newton method in the main text and further discuss it in the appendix.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Reproducing our algorithm is simple by following the steps we outline in Algorithm 1. We also include code in the supplementary material to showcase a simple implementation of our method using Stable Diffusion.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The models (Stable Diffusion) and datasets (ImageNet, WikiArt, PartiPrompts, FFHQ) used are all publicly available. Furthermore, we include the code to reproduce the algorithm the supplementary material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (ittps://nips.co.
public/guides wwwaa//eee/odeeetmmmmsiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https?
7 cttpssiic gui CodeeSumeeeeeeeeeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We describe in detail each experiment we performed, including the exact data splits and how sampled from them. We also describe in detail the hyperparameters chosen for our algorithm.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

Justification: It is not common in related works on training free constrained sampling from generative models to report error bars on experiments regarding the generated image quality. Following the previous work, we only performed our experiments once.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the compute setup we used and describe the inference time and memory requirements for running our and previous algorithms the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
htw.h/eeeeiiiiddeeeeeellllllllle ines?
Answer: [Yes]
There were no human subjects our tests and all datasets used are publicly available.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
17

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We included a paragraph discussing the societal impacts of our work in the
appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justífication: We release no new models or datasets. The existing models we utilize (Stable Diffusion) include such safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification: All models and datasets we used are publicly available and open for research use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper swi thcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No human subject experiments were performed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
No human subject experiments were performed.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs were used.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Further analysis of the proposed algorithm
A.1 Spectra of the denoiser Jacobian
In the main text, we resorted to inexact Newton methods 7 to analyze the convergence of the proposed algorithm. We reiterate that the original Newton (or Gauss-Newton) method aims solve the system
Je, =.
(16)
For general discussion on how to solve this system of equations we refer the reader to Gavin [2] In our case of denoising diffusion, where computing the inverse of J is expensive, inexact Newton methods utilize an approximate solution et to Eq (16), which is guaranteed to converge if the residual strictly reduced every step
||ll
= e Je,
n, E (0,
(17)
|el|
When substituting the proposed update ex = >Je (Eq. 10
we get
r ll |ell2
(1 - eel2
||
XJ
|2llell2
||I 1--2122 < <.
(18)
lell2
Therefore, we need to show that the spectral norm of the matrix I - XJ² is strictly less than 1 for correct choice of learning rate A.
J was diagonalizable (or generally normal), then we could directly estimate || - XJ using the largest eigenvalue of J. While cannot directly assume that J is normal (we explicitly mentioned that is not guaranteed to be symmetric), we know that should be "almost' symmetric, as is the optimal denoiser solution shown in Eq. (15). This means that could express J = 5 K, where S = (J +
+J") J is the symmetric and K = (J J the skew-symmetric component. Since both the symmetric and skew-symmetric components normal we can estimate their spectral norms. In the case where || |||| < [S||,, which we expect since J is 'almost" symmetric, we can
approximate I XJ
| 12 Ro | À max 1
(19)
where μ4 are the eigenvalues of S.
To get estimate of the magnitudes of the largest eigenvalues of S, which are real, and of the eigenvalues of K, which are imaginary, we use the Arnoldi iteration [[] For the Arnoldi iteration we only require access to the matrix-vector products Jo and J v, which can be computed using the approximation of Eq. [14 and backpropagation respectively.
Running the Amoldi iteration algorithm J. (J + J and (J-J") (J-J we plot the magnitude of the largest eigenvalue of each matrix Fig. 6 (a). We see that the contribution of the symmetric part of the matrix is the strongest, validating assumption that the diffusion model's 's Jacobian J is "almost' symmetric.
If we approximate the spectral norm of Eq. 18 using the symmetric component of the Jacobian we see that for correct choice of we can ensure that | À max; p4d < 1. Now we revisit the example of Fig- F where we inpaint half of the image. Instead of using the empirical learning rate used in the paper (linearly decreasing and gradient normalized by the co-norm) we compute the largest eigenvalue of (J + J2)/2, p. and select different À so that we satisfy violate the bound provided by Eq. (19).
In Fig. (b) we show that the error is not reduced for learning rate values that consistently violate the proposed bound. As expected, our algorithm consistently reduces the error every iteration for small enough learning rate, where the bound is always satisfied. When we use the empirical learning
rate, shown in Fig. 6 (c), we see that the algorithm bounces between satisfying and not satisfying the computed bound and ends up at lower error than the constant learning rate of Fig. 6 (b). We
posit that the normalization by the oc-norm helps the learning rate adjust the step size such that substantially reduces the error while not leading divergence.
21

---

cigenvniuc
Error convergence bound the
symatzlric and sume-ryumetire componeuts
Error for different learning
leazning rale used the paper
Max eigrennloe inggritate
[t ALP
Encrr
Auf 1-11
-x- P Audde
[I
184
620
020
100
Thoetap
Txastp
(8)
(b)
Figure 6: Using the proposed analysis we visualize in (a) the largest eigenvalue of J as well as its symmetric and skew-symmetric components. The largest eigenvalue follows closely the largest eigenvalue of the symmetric part of the matrix. In (b) we demonstrate the convergence of our method for different learning rates. (c) we show that our adaptive learning rate scheme initially oscillates around the theoretical convergence bound and eventually settles well-below it. These initial oscillations may be important for the high high-quality results, as using lower learning rate did not attain similar-quality images.
0 ki
0xp
t 900,
r= 0.89
0.04
700,
r 0.50
400,
0.56
-0.04
0x
0x,
-0.04
Figure 7: Sample pairs (i, (ks, 1) of the denoiser Jacobian Vn, 20 Vn, F F[araaare]] for different timesteps t. We observe that the Jacobian is not symmetric, justifying the difference between the proposed update steps and the previously used gradient descent steps.
A.2 Symmetry of the Jacobian and difference in updates
Symmetry Although, theoretically, the Jacobian of the denoiser should be symmetric (Eq. 15). the trained diffusion model does not exactly match the real score and can yield non-symmetric Jacobians. In Figure 7 we perform a simple experiment to visualize this difference; we select a random image from the ImageNet X validation set and employ the Stable Diffusion 1.5 model (3 to denoise three different noise levels {900, 700, 400). We encode the image using the VAH encoder, scale and add appropriate noise get the intermediate diffusion latent for each timestep. We then give the noisy image the denoiser network and compute the gradients dá /0x)," and 02))") /Ox k.l for randomly chosen pixels (i, (ks,) 1) using backpropagation. When we plot the gradients and compute correlation coefficient r' we observe that the values deviate from y = x, which would indicate symmetric Jacobian.
Toy experiment Having shown that the Jacobian is not symmetric, we expect to find differences between the gradient updates of gradient descent (Eq. 2.2 and our proposed update (Eq. 10.. To highlight these differences, we set up toy experiment where update an identical initial image with the two different directions, under the same condition. The experiment is showcased in Figure g and again utilizes the Stable Diffusion 1.5 model. We create synthetic black-and andk-ind grid image
22

---

Sample
Condition
J'e Je
(b)
Figure 8: We present a simple experiment compare the proposed gradient update Je with that of previous methods J The goal is to update a noisy version of the image shown in (a) at f = 800, such that the diffusion model's prediction of the final image includes a red square the middle, shown in (b). The results of gradient descent in (c) and our gradient update in (d) demonstrate that updates performed by previous methods lead considerably different result than the update propose in this paper.
Sample
Condition
Ours
Gradient Descent
Figure 9: Further runs of the toy experiment A.2
(Figure (a)), which we blur, deterministically encode, scale and noise to get diffusion latent at 800. We then set up constraint where we add red square the center the original image (Figure (b)). Instead of decoding the image and applying the constraint in image space, we also encode constraint image and apply directly in the Stable Diffusion latent space.
We run gradient updates with the same learning rate of À = 1 for each and demonstrate the final predicted 2p for the J update of Eq. (2.2) (a) (Figure g (c)) and the proposed Je of Eq (10 (Figure [8) (d)). The final result shows how the model intends to change the entire image when asked add a red square in the middle.
The resulting images differ substantially, with the proposed direction producing a more coherent image that tries to copy the newly introduced texture to the correct locations, i.e. the intersections of the lines. Although this is an empirical observation, we hypothesize that the two different directions have vastly different effects the image. In Figure we repeat this experiment multiple random seeds.
More visualizations In the main text, referred to the difference between J and J as motivating factor for our approach and measured that difference by comparing elements (i.j) and (j, i) of the Jacobian matrix (Figure 7). In Figure I0 we extend Fig. 2 We visualize the difference
23

---

J
g
9000
Xy 800
% 600
Xy 400
We visualize the difference between J and J by computing how the matrix wants to change the entire image would look for a perturbation of a single pixel. By ( we denote this exact change, which corresponds the sum of 4 columns in the matrix (4 latent channels per-pixel). Our proposed update direction better captures longer-range dependencies by better maintaining shapes. Even though we use numerical approximation, the proposed direction is sharper in regions, like the outline of the cat.
Figure 10:
between using the Jacobian and its transpose by numerically computing how the Jacobian and the Jacobian transpose of the entire image would look for single pixel, i.e. which parts of the image affected by change in single pixel.
In Figure 10 we simplify the notation and denote as (6) the Jacobian for pixel i, which we compute by summing the squares of the matrix columns that correspond to this pixel's latent values. If our approach were equivalent to backpropagation, i.e. J J then a change in single pixel would have similar effect the rest of the image, up some noise because of the finite difference approximation.
Contrary to expectations, we find a significant difference between our proposed direction and backpropagation, with our approach having a better effect on retaining shapes and symmetry across the image. We see that in many cases, the model is trying to change correlated parts of the image together, i.e. change both eyes or ears simultaneously. showing us some of the knowledge that the model has acquired regarding the image space in general through its text-to-image training.
A.3 Inexact and exact Newton
We set up a simple experiment on MNIST to compare the proposed inexact Newton update step with the exact Newton, i.e. computing the inverse of the Jacobian. We train a 25M parameter diffusion model, following the architecture of [9, on 32 x 32 zero-padded MNIST images. For this model we use intoodifferrnniationn to compute the 1024 1024 Jacobian matrix, which takes 10 seconds on our GPUs.
We randomly sample images from the training dataset, add noise corresponding to L= 700 and = 500 and denoise, predicting the d.. Then, we aim apply a simple edit to the predicted a that increases decreases the value of single pixel in the image. For that edit, the error corresponds
24

---

Je
J'ce
J'c
Je
J'le
J'c
Predicted Xo
T pixel value
3
f
3
3
Update Direction Result
f
9
0
0
0
t-700
Je
F'e
J'e
Je
J'e
J'e
Predicted xp: 1 or pixel value
O
6
5
O
5
2
2
2
Updato Direction
/
/
3
1
3
+
4
4
3
3
3
Result
5500
Figure 11: We show the difference between the inexact Newton, exact Newton and gradient descent updates on simple setting of generating MNIST digits. We simulate constraint by choosing to increase or decrease the intensity the inexact and exact Newton methods are similar, with the inexact approach requiring much less compute and being easy to compute practical settings.
single pixel the image. Qualitatively, the updates made with
+1 or-1 in the location of the edit. For each specific edit, we compute the update we should make to the noisy i; using our inexact Newton method Je, the exact Newton J-'e and gradient descent J For the exact Newton method the Jacobian can be ill-conditioned, requiring utilize pseudo-inverse in computing the update J-le.
showcase the results of editing the image with the three update directions, The exact In Figure II
Newton step makes definite steps in updating the image, which converge faster to the desired solution. Our inexact step, qualitatively attempts make similar edits to the image; e.g. when increasing the pixel intensity inside the digit '0', both updates also delete parts of the side, turning the digit into 5 (Figure IT] left). Of course, discussed the main paper, our inexact step requires multiple, smaller steps to achieve the result, but is much cheaper to compute. Finally, the gradient descent direction gives similar results
proposed update, but as discussed in A.2 showcases qualitative differences that we hypothesize arise from the imperfect training of the diffusion model.
A.4 Inexact Newton steps in VAE space
In Section 4.2 described how instead of following the Newton recipe, which requires the inverse of the constraint Jacobian J used the gradient descent direction J when dealing with non-linee
25

---

Generated Image Backpropagation
Generated Image Newton
Downsampled Image
Image
Figure 12: Comaprison between using backpropagation and Newton steps with linear constraints in image space.
constraints. Here, we discuss a special case of non-linear constraints, constraints that are linear in image space. These constraints still non-linear for latent diffusion models (27] since they involve the VAE decoder, which non-linearly transforms the latent representations that the diffusion model generates RGB images.
However, when the constraint is linear image space we find that we can still utilize an inexact Newton approach to avoid backpropagation through the decoder. More specifically, for the case where
C(æ,)) (AD( - (AD(((((z))
(20)
we write the first-order Taylor approximation
C(æ: et) = AD(āū((z)) AJpJe, - - (AA((2((()) AJpJe: -.
(21)
When we solve for Ve, we get
- y) y) = J.JJe.
(22)
Similarly the results in the main paper, assuming that inverses exist, end up with the following system
A T (AD(20((z)) - -y) = Jp-ee
(23)
which we rewrite as
e; Jo.
= A==A"(AD(((((rr) - y).
(24)
To apply the proposed Newton approach and get an update direction for at, we must first solve the system e; = b for b, and then use the approximate solution e, = Jb. In the super-resolution experiments we performed the paper, we opted for the gradient descent approach, which can be expressed as b = Jffe: However, this requires backpropagating through the decoder model which in some cases may inefficient or altogether unavailable.
As an alternative, we can again resort to inexact Newton and use an 'approximate" inverse to Jp. the encoder Jacobian Jg. Intuitively, the encoder model performs the inverse operation of the decoder. and therefore we could employ it to 'invert' the decoding operation. Using the encoder allows us to replace backpropagation with forward passes by
Jge, (E(D().) + de,) E(D(
(25)
By combining the Newton step for the VAE space and the Newton step the denoiser we can run our provide some qualitative
inference algorithm with no backpropagation operations. In Figure 12
26

---

Table 6: Ablation study on the choice of 8 and exact forward-mode auto buto-differentiation. K' is the number of steps, the learning rate and the finite difference step size. By * we denote the parameters used for the experiment in the main paper.
Inpaint (Box)
Parameters
PSNR t
LPIPS
FID |
Time
K 5, À 0.5, 0.0005
17.47
0.37
69.02
158
K = 5, À = 0.5, = 0.005*
18.30
0.30
42.01
158
K 5. À 0.5, 0.05
18.32
0.31
42.44
15s
K 5. À = O.5, = 0.5
15,80
0.34
61,40
15s
K 5. À 0.5, exact
18.27
0.31
44.90
76s
comparisons of super-resolution with a Stable Diffusion model when using backpropagation through the decoder and the inexact Newton step in VAE space. Nevertheless, we opted for backpropagation in our experiments since the time required for multiple forward passes through the encoder and decoder models ended up being the same backpropagating once through the decoder, and the memory requirements were not prohibitive. In cases where memory is an issue, the 'pure' Newton approach could be an appealing alternative avoid backpropagating through the decoder model.
Using the the Jacobian of the encoder as an approximation for the inverse Jacobian of the decoder, has been discussed before in the context of autoencoders in Sorrenson et al. [36) To our knowledge, we are the first to employ this approximation for the diffusion autoencoders.
A.5 Finite-difference approximation and exact gradient computation
To compute the proposed update Je, we the finite difference approximation Je (f( + +e) (x)))/ In comparison, the gradient descent direction J e is exactly computed using automatic differentiation, usually implemented the backward gradient computation, c.g. e. backward in PyTorch.
Some libraries also offer forward-mooe auto-differentiation, which directly computes the Jacobian- vector product Je. However, forward differentiation is not always implemented or optimized as well as the backward propagation of gradients. In the case of Py ToTcch, which is what use for running our experiments, forward mode differentiation not directly implemented for many of the custom layers of the Stable Diffusion model.
To perform a comparison between our finite difference approximation and the exact forward computa- tion, we resorted to the double backward trick, which computes the forward-mode gradient with two backward calls (tor ch.ch... . autograd. functional. jvp () in PyTorch). This is course expected to be slower and more memory-intensive, but difte-difference approximation employed in the paper. For completeness, we also ablated the choice of the step size 5. We repeated the box inpainting task of Table 3 and present the results in Table 6
use it as baseline to verify the validity of the
In our ablations we find that the finite-difference approximation is robust to the choice 5, and only fails when using too small (0.0005) and too large (0.5) values, and (ii) performs as well as the exact forward mode dittediiferentiationn while requiring only fraction of the time.
A.6 DDIM and other sampling methods
To apply the proposed method, modified the DDIM sampling algorithm [33) We provide side- by-side comparison to show the difference between the original DDIM and the proposed algorithm. Extending algorithm to other sampling methods should be intuitive by alternating between gradient updates from our algorithm and the diffusion updates computed with the diffusion sampling algorithm used. In Algorithms 2 we sketch out pseudo- algoorthh for applying the proposed algorithm to any diffusion solver.
Beyond DDIM, we also implemented our method with the PNDM scheduler 19 where we get results indistinguishable from DDIM. Both DDIM and PNDM implementations are provided in the
GitHub repository In Appendix B.4 where we applied our method rectified flows, use the
27

---

Euler ODE solver. Again, our method is intuitive to apply by interleaving the diffusion updates with our proposed optimization steps.
Algorithm Pseudo-algorithm for sampling us- ing solver and pre-trained diffusion model.
Algorithm 3 Pseudo-algorithm for constrained sampling with solver, pre-trained diffusion model and using the proposed algorithm.
1: Input: Pre-trae Pre-trained diffusion model zo(xz), diffusion Solver(), diffusion steps t1, 12...., ... diffusion schedule a;
Pre-trained diffusion model (0(x))) diffusion Solver(), constraint C(zo, y)
Input:
2: 21 I))
[ xx) yll condition y. step size iterations K, learning rate X, diffusion steps
3: for t = h1 11,1t....,t... 1 do
4:
I)
f1, t2. ........,.... diffusion schedule a; 2: Φ1 ~ (),
5: Lt+1 = (olver((g 2e, a,, t 1)
6: end for
for t,,t...... ty-- do
7: Return: æN
4: for i = 1.2...., K do (J (Z
y) E, = (0 + òe) to (0:)))))/
6:
7:
Ct 11 dei
8:
end for
9:
21 I.
10: I(+1 = Solver (x))) z2, au,t + 1)
11: end for
12: Return: NN
B
Additional results
B.1
Mask-guided generation
For the mask-guided generation experiment utilized pre-trained face segmentation model from huggingface htttss:/huugiigeee
jow.onnannnniin httpo//aaaaaaaaaaaaaaaaaaiiiiiia parsing. In Figure 13 we provide qualitative results of our segmentation mask-guided generation experiment described in the main text. The quantitative results are provided in Table il the main text.
B.2 Number of steps and learning rate ablation
We repeat the ImageNet box inpainting experiments with fewer optimization iterations and higher learning rate. We show qualitative results in Figure [4 where we observe that running fewer optimization steps gives blurrier results, which expected as the known regions the image also seem to not have converged to the given values. Using higher learning rate leads to the model sometimes 'overshooting' by inpainting the missing regions with realistic-lookigg parts that do not necessarily fit the rest of the image. The guantitative results are presented in Table in the main text.
B.3 Non-difflerentaabl constraints
There is no restriction on defining the constraint C as long as we can get the direction e towards which want to push the image mn. In the non-linear constraint case, resorted to using the gradient descent direction J (f(2o(z)) - y) to avoid computing the inverse Jacobian of f. In theory, any direction e that locally minimizes the constraint can used with the proposed algorithm.
As toy example, we generate images with pixel values quantized to be either 'on' or 'off' (-1 or 1). The constraint first measures whether pixel value is positive or negative and then sets error direction to æ|| for which we can easily compute a local gradient that reduces the cost C. Using the prompt photo of cat', generate quantized images shown in Figure 15
_ for every pixel accordingly. This is a non-differentiable constraint
28

---

Ours
Reference
Generated
Reference Mask
Generated Mask
MPGD p=1
Reference
Generated
Reference Mask
Generated Mask
MPCD p=0.5
Reference
Generated
Reference Mask
Generated Mask
Figure 13: Examples segmentation guided generation using our method and MPGD.
B.4 Time-distilled models
We test whether our method works on distilled models such as rectified flows 20 and consistency models [35]. These techniques reduce the number of inference steps by distilling from base diffusion model. Our method does not explicitly depend on the number of inference steps used, the type of model or the noise schedule; the only requirement is having way to estimate the final clean image from the current step, which both rectified flows and consistency models admit. Thus, applying to rectified flows and consistency models is intuitive.
We employ our method to inpaint images using the 2-rectified flow model distilled from Stable Diffusion, InstaFlow [21), using inference steps. We observe that although our algorithm still works
29

---

Image
n= 1r=
n=2, 1r=0.5
n=5, 1r=1
Ground Truth
Figure 14: Examples of box inpainting when using different optimization steps and learning rates.
Figure 15:
We apply a non-differentiable constraint to generate quantized images. Using the prompt a photo of cat' we generated binary images of cats.
with rectified flow and just 5 inference steps, we do consistently get high-quality samples did with Stable Diffusion when using the same hypeparameters and it requires more optimization steps. Ultimately, we utilize line search to find the À for every gradient update we perform. This increases inference time further, mitigating the gains from using time-distilled model.
We hypothesize that using rectified flows (or any distilled model with fewer steps) may be more challenging since the initial noise dictates most of the content the final image. Therefore, the first optimization steps perform must get sufficiently close to the correct solution.
When using diffusion model, we can get away with imperfect optimization steps as there is more room for fixxin'' the image in later timesteps. We show examples of the rectified flow inpainting in Figure T6 We also refer the reviewer to Figures 17 18. where we show the intermediate generation steps for Stable Diffusion.
30

---

Image
Rectified Flow
line search
Rectified Flow no line search
Ground Truth
Figure 16: Using a rectified flow model 21 that generates images in just steps requires more careful optimization steps, there is less room for errors during generation.
Image
10000
t=950
t=900
t=550
t= 750
t=700
t=650
t=600
t=550
t=500
t=450
t=:00
t=330
t=300
t=
t=200
t- 150
t- 100
t-50
Result
Ground Truth
Figure 17: Visualization of the intermediate inference steps for the inpainting task.
B.5 Inference visualizations
In Figures 17 18 and 19 we visualize the intermediate steps of the proposed algorithm for the inpainting, super-resolution and style-guided generation tasks respectively. Our method quickly converges to a plausible image and then further refines it to better satisfy the constraint over the diffusion timesteps. For style-guided generation, first few initial steps before the specific style provided applied.
that the structure of the image is defined in
B.6 Efffet of convergence speed final images
ask the question of how does convergence speed affect the quality of the generated images? Previous works found that applying high weight on the constraint led to unwanted artifacts in the generated images [5). We hypothesize that, apart from artifacts the gradient, 'over-optimizing' for the condition given timestep affect the generation quality. In practice, if push the initial
31

---

Image
t= 1000
t=950
t=900
t=550
t= 800
1= 750
t==00
1==50
t==600
t=550
t=500
t-450
t-350
t-300
t-250
150
t=50
Result
Ground Truth
Figure 18: Visualization of the intermediate inference steps for the super-resolution task.
2t too far from the inputs the denoiser network is expecting, either with high weight on the gradient update or by performing too many updates, we should be seeing non-realistic images in the output.
We investigate this by running the same inpainting experiment with optimization steps per timestep (Figure 20) and 20 optimization steps (Figure 21). Although expected see difference in the final generated images, we find that both converge to similar quality results. Our proposed optimization steps single timestep consider the Jacobian of the denoiser model, which find acts as 'regularization' and makes difficult to produce X, inputs that satisfy the condition 'early'. Even when running the optimization for more steps single x,, we that although the sample converges faster to the desired condition, the denoiser is still able to continue the diffusion process of Xt.
B.7 More Qualitative Results
In Figure 3 we provided qualitative results on free-form inpainting and super-resolution. In inpainting, our model consistently performs as well as the slowest bascline, P2L. For super-resolution, P2L which also infers prompt seems to generate better-fittin fitttr- textures for the images. We hypothesize that by also inferring a prompt the high-frequency detail generation better-guided in the super-resolution task. In contrast, in inpainting, the non-masked pixels contain enough information about the textures that need to be placed around the image.
In Figure 22 we showcase additional results on the box inpainting task. MPGD [3), which does not backpropagate the constraint error through the diffusion model completely fails at inpainting missing region. We attribute that to the minimal ability to influence pixels that are 'far' from the constraint at lower noise levels without probing the model weights. The Naäve algorithm replaces the known pixels in the estimated final image at every denoising iteration.
32

---

[15] controls the influence of the text prompt on the Style Image
t=1000
1=950
t=900
t=
800
1=750
t=000
t=650
t==60
t=550
t= 5500
t= 150
100
t=350
t=300
t=250
t=
Result
Prompt
boy going school
Figure 19: Visualization of the intermediate inference steps for the style-guided generation task.
In Figure 23 present additional results style-guided text-to-image generation for single prompt. Qualitatively, we see that the style of the images generated with our algorithm better matches the style of the reference image, even when using different model to define style (OpenCLIP). In Figure 24 we show images generated with different styles and text prompts. Here, we show how increasing the classifier-free guidance weight final generated image.
C Societal Impact
The work presented in this paper aims to advance the field of machine learning, specifically generative modeling. Solving constrained sampling tasks with generative prior, can greatly benefit from the better utilization of the image prior. One specific domain is compressed sensing medical imaging, where generative priors like diffusion models have been used to reconstruct low-dose CT scans and accelerated MRIs. We leave to future work the application of the proposed algorithm to these settings.
However, we acknowledge that there are potential societal consequences of our work that can have negative impact. The one highlight is the ability to edit images with the intent to deceive and mislead. While it is true that existing models can already be used to alter images, we understand that our work could offer more precise control over the generation and lead more convincingly fabricated content.
33

---

5 steps
Image
1000
t-950
t=900
t-850
750
t=700
650
t=600
550
500
1=4
1=400
15=350
1:=300
1==250
1=200
t= 150
t= 100
t=50
Result
Ground Truth
Figure 20: Convergence for the inpainting task when using K 5 optimization steps.
34

---

20 steps
Image
t=1000
1=950
1==00
t=850
t=800
750
1=
1=650
t==600
t=550
15000
Wt
0 U
orw
SUT UIF
am1
0
1=450
1=400
1=350
1=250
X un
UTI
umm
Xln
150
100
t=50
Result
Ground Truth
un
umm
um
am
Figure 21: Convergence for the inpainting task when using 20 optimization steps.
Moskal Innege
LIPSS
PSUD
FreeDeed
MPGD
Naive
Impiittnn Model
Our.
Ionge
Figure 22: Box inpainting examples for all methods. Naive replaces the pixels with their true values + noise during inference.
35

---

glasses"
Style
(OpeaCLIP)
FreeDoM
MPGD
Figure 23: Examples of guided text-to-image generation for single prompt and multiple styles.
Ours
Text Prompt
Style
Ours (CLIP, w=2)
Ours (CLIP, w=5)
(OpenCLIP. w=5)
Fre:DoM
MPGD
ta box"
close-up of the
eyes of an thol"
old with a long grey beard and green eyes"
"a comic about fother and playing termii"
Figure 24: Examples of text-to to-image generation for multiple prompts and styles.
36


Reviews:
[{"review_id": "ZoAl0DTyGU", "reviewer": "Reviewer_WTsj", "summary": "The paper introduces a fast, training-free algorithm for constrained sampling in pre-trained diffusion models, enabling high-quality image generation under arbitrary linear and non-linear constraints. By approximating Newton optimization steps using only forward passes through the denoiser, the method avoids expensive backpropagation while achieving results comparable to or better than state-of-the-art approaches. The technique is demonstrated on tasks like inpainting, super-resolution, and style-guided generation, significantly reducing inference time and memory usage.", "questions": "NA", "limitations": "NA", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength: The  method proposes a new perspective for diffusion inverse problem to utilize the constraint signal besides backward gradient calculations.\n\nWeakness:  My major concern is that the proposed method sacrifices performance to gain computation efficiency, it will be good to see the quantitive comparison of using newtown approximation method and using exact gradient descent, not just showing theoretical error analysis or visual comparison. Also SD1.4 has been three years old, would be good to use more recent backbones.", "was_revised": true, "final_justification": "After the discussion, I hold my opinion of borderline reject and open to any final decision", "rebuttal": "We want to thank the reviewer for their comments and address their concerns below:\n\n**Sacrificing performance for efficiency**: \n\nWe would like to point out that there is **no significant drop in performance** when using our method, while it requires just a fraction of the time to run. In Tables 2 and 3, we show that our algorithm is consistently superior to existing training-free approaches, especially in the more challenging box inpainting task, where it is only outperformed by the fine-tuned SD-Inpaint model, which required an additional 100k iterations of training.\n\nRegarding Table 1, although our algorithm performs worse than the P2L baseline in 8x super-resolution, P2L requires 30 minutes to run on a single image (vs 1min for ours) and also uses the PaLI VLM to caption the low-resolution images before the diffusion inference. As briefly mentioned in lines 207-208, we believe that the main advantage of P2L comes from introducing text conditioning, whereas all previous methods relied on the unconditional model. To verify this assumption, we ran the x8 super-resolution with text prompts generated from the downsampled images using Qwen-2.5 [Qwen] as the VLM. The results we present below show that we can indeed bridge the performance gap and even improve upon the PSNR and FID metrics when introducing text captions to the super-resolution process. We will add this result and relevant discussion to the paper to complement Table 1.\n\n| Method | PSNR ↑ | LPIPS ↓ | FID ↓ |\n| - | - | - | - |\n| P2L (with VLM captions) | 23.38 | 0.386 | 51.81 |\n| Ours | 24.26 | 0.471 | 60.99 |\n| Ours + VLM captions | 24.95 | 0.405 | 44.74 |\n\nTo summarize, our method is both substantially faster and performs as well as or better than the baselines. There is no notable trade-off between efficiency and performance.\n\n**Approximate Newton vs gradient descent**: \n\nThe goal of training-free algorithms is to compute and apply a gradient during the diffusion sampling that 'pushes' the generated image towards some desired condition. The previous methods we compare to in the paper (P2L, LDPS, PSLD, FreeDoM, MPGD) all use gradient descent to compute the update direction. Gradient descent is expressed as $J^Te$, where $J = \\nabla f(x)$ is the denoiser Jacobian and $e$ the constraint error.\n\nWe propose an alternative way to compute the update direction, based on approximating the steps a Newton method would make. The *exact* Newton method would use $J^{-1}e$ as the update direction, but this requires computing the inverse of the Jacobian, which is infeasible for these large diffusion models. We show that we can adequately approximate the Newton step with smaller updates in the direction of $Je$. This has been applied before in numerical optimization, as discussed in *Inexact Newton Methods* [5].\n\nTherefore, the comparison between our approximation to Newton's method and the exact gradient descent is shown in the experiments of the main paper. The previous methods using gradient descent are slower and produce worse results than the proposed approximate Newton method.\n\n**Approximate Newton vs exact Newton**: \n\nIf the reviewer was referring to the accuracy approximating Newton's step $J^{-1}e$ with multiple inexact steps $\\lambda Je$, unfortunately, there is no practical way to quantify it, as computing $J^{-1}$ for models at the scale of Stable Diffusion is prohibitive. In Equations (10)-(13), we discussed why our approximation converges to the correct result, but we are unable to directly compare the two update directions for the tasks shown in the paper.\n\nRegardless, we ran a smaller-scale experiment on MNIST where we can easily compute $J$ and $J^{-1}$ and show that both exhibit very similar structures. This aligns with the intuition we provided in lines 133-134 for using the proposed approximation. In the interest of space, we refer the reviewer to our response to Reviewer NFw9 for more details on this additional experiment.\n\n**Finite-difference approximation vs exact gradient**: \n\nTo avoid any confusion regarding the term **approximation**, it's important to note here that to compute the proposed direction $Je$, we employ finite differences for the Jacobian-vector product $Je \\approx (f(x + \\delta e) - f(x))/\\delta$. This is also an approximation, but for the Jacobian-vector product computation and **not** the constraint gradient update direction.\n\nThe baselines that utilize gradient descent use automatic differentiation/backpropagation through the network. Our update can also be computed using automatic differentiation (forward-mode automatic differentiation). However, forward differentiation is not always implemented or optimized as well as the backward propagation of gradients. \n\nFor completeness, we ran an ablation to study the choice of $\\delta$ for the finite difference approximation and compare it to the exact gradient computation. For the forward gradient computation, we used the double backward trick since the forward gradient is not implemented for all Stable Diffusion layers in PyTorch. We ran the box inpainting task of Table 2 and found that our method (i) is robust to the choice of $\\delta$, only breaking when using very small (0.0005) and very large (0.5) values, and (ii) performs as well as the exact Jacobian-vector product computation while requiring 1/5th of the inference time.\n\n| Method | PSNR ↑ | LPIPS ↓ | FID ↓ | Time |\n| - | - | - | - | - |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.0005$ ) | 17.47 | 0.37 | 69.02 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.005$ ) | 18.30 | 0.30 | 42.01 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.05$ ) | 18.32 | 0.31 | 42.44 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.5$ ) | 15.80 | 0.34 | 61.40 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, exact forward diff ) | 18.27 | 0.31 | 44.90 | 76s |\n\n $K$ number of steps, $\\lambda$ learning rate, $\\delta$ finite difference step size\n\nThus, using finite differences to compute the Jacobian-vector product is much faster and does not incur a drop in performance.\n\n**Using Stable Diffusion 1.4**: \n\nStable Diffusion 1.4 has been widely used to benchmark training-free inference algorithms. Therefore, we also chose this model to showcase the advantages of our proposed method against the existing baselines. The proposed algorithm is not constrained to Stable Diffusion 1.4; we showcase qualitative results on a rectified flow model in the Appendix and provide an implementation for Stable Diffusion 2.1 in the supplementary code -- the results we get with SD-2.1 are similar to SD-1.4, but we did not perform a full-scale evaluation since all the baselines are evaluated only on SD-1.4.\n\n[Qwen] Bai, Shuai, et al. \"Qwen2. 5-VL technical report.\" arXiv preprint 2025"}, {"review_id": "pbL5CONxyt", "reviewer": "Reviewer_uiuT", "summary": "This paper presents a novel algorithm to enhance the efficiency and quality of constrained image generation using pre-trained diffusion models. The authors identify a key trade-off in previous works, methods utilizing backpropagation are often slow and memory-intensive, while faster approaches that avoid it can fail to capture long-range dependencies. To address this issue, the proposed method approximates a Newton optimization step to apply constraints, avoiding computationally expensive backpropagation. The paper validates this approach on various linear  and non-linear tasks , demonstrating competitive performance in most cases.", "questions": "N/A", "limitations": "See weaknesses", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The primary strength of this work lies in its computational efficiency. By approximating a Newton step using two forward passes instead of backpropagation , the algorithm effectively reduces inference time and memory requirements, making the approach highly practical for real-world applications.\n\n- Despite its speed, the method produces high-quality results that are competitive with, and in most cases superior to, state-of-the-art training-free methods. \n\n**Weaknesses**\n\n- While the results are strong overall, the method's quantitative performance is not uniformly state-of-the-art. For example, in the 8x super-resolution task (Table 1), the reported FID score for \"Ours\" (60.99) is noticeably worse than that of the P2L baseline (51.81). An analysis of why the proposed method is less effective in this specific scenario would strengthen the paper.\n\n- I noticed minor artifacts in some of the qualitative results. Specifically, in the second row of Figure 3 , the generated image of ours appears to have a glyph watermark. Could the authors explain this phenomenon? An explanation of the potential cause (e.g., artifacts from the pre-trained model or the optimization process itself) and possible mitigation strategies would be beneficial.", "was_revised": true, "final_justification": "The authors have addressed my concerns and I remain my score of Borderline accept.", "rebuttal": "We would like to thank the reviewer for taking the time to read our paper and provide valuable feedback. We respond to each of the weaknesses mentioned below:\n\n**Quantitative performance is not uniformly state-of-the-art**:\n\nIn Table 1, we showed that the P2L baseline performs better on x8 super-resolution compared to our method. However, we point out that:\n1. P2L requires 30 minutes to run on a single image, severely limiting its usability. Our method only requires 1 minute.\n2. P2L uses a VLM (PaLI) to caption the low-resolution image before running the image sampling algorithm. We believe that for super-resolution, the text becomes integral in synthesizing the correct textures in the given image, which we hypothesized was the main advantage of P2L as mentioned in lines 207-208. Since all other previous methods only utilized the unconditional model, we also resorted to unconditional diffusion sampling in the main paper.\n\nTo offer a complete picture, we verified the hypothesis that the advantage of P2L comes from using VLM-generated text prompts. We ran our 8x super-resolution algorithm with captions generated from an open-source VLM (Qwen-2.5 [Qwen]). For each low-res image, we first captioned it with the VLM using the instruction \"Describe this image\" to obtain a text prompt, and then applied our algorithm, conditioning the diffusion model on the VLM-generated prompt.\n\nThe results presented below show that we improve upon the super-resolution PSNR and FID metrics of P2L when we use text captions with our proposed algorithm. When the text is correct, it accurately guides the generation of high-resolution details in the image. We observed that most current VLMs (GPT-4o, Qwen-2.5) are fully capable of providing meaningful captions even with low-resolution images.\n\n| Method | PSNR ↑ | LPIPS ↓ | FID ↓ |\n| - | - | - | - |\n| P2L (with VLM captions) | 23.38 | 0.386 | 51.81 |\n| Ours | 24.26 | 0.471 | 60.99 |\n| Ours + VLM captions | 24.95 | 0.405 | 44.74 |\n\nWhen it comes to comparing the various training-free sampling algorithms for super-resolution, adding a VLM to the pipeline introduces uncertainty, as the evaluations also need to consider the quality of the VLM and the specific prompt(s) used. Thus, we will include both results (super-res with and without text prompts) in the main paper and Table 1 for completeness.\n\n**Artifacts in generated images**:\n\nThe artifact in Figure 3 row 2, is a watermark hallucinated by the model. This is solely an artifact of the image prior of Stable Diffusion, which was trained on (often watermarked) web images. We decided to use this image for Figure 3 because we found it interesting that our algorithm not only follows the conditions closely but also hallucinates novel high-frequency details under the constraint.\n\nHere, the hallucinated watermark is adhering to the imposed constraint. The mean squared error between the downsampled watermark image and the constraint is around 0.0008, which is also the mean squared error we get when we compare the constraint to another generated lighthouse image that has no hallucinated watermark. In contrast, previous methods often produce blurry results that lack hallucinated high-frequency textures, which the image prior should induce in the generated samples. For instance, all other methods in Figure 3 row 2 produce blurry textures for the rocks.\n\nTo avoid confusion, we will replace this image with another sample that has no watermark hallucination, and move this example along with the discussion to Appendix **B.7 More Qualitative Results**.\n\n[Qwen] Bai, Shuai, et al. \"Qwen2. 5-VL technical report.\" arXiv preprint 2025"}, {"review_id": "H19TPD4dHR", "reviewer": "Reviewer_NFw9", "summary": "This paper proposes a novel method for training-free constrained sampling. Specifically, instead of backpropagating through the score function (which evaluates whether the constraints are satisfied) and the denoiser, the paper uses an approximate Newton optimization step to avoid backpropagation (though this comes at a cost of one more feedforward evaluation).\n\nEmpirical evaluations demonstrate improved performance in image inpainting. It is also faster than some baselines.", "questions": "- How does the performance change w.r.t. $\\delta$?\n\n- It seems that the proposed method should have similar runtimes compared to the baselines if given the same number of steps. I wonder how the performance compares against the baseline if given the same computation budget?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is training-free and can be adopted to condition on linear and non-linear constraints. The experiments demonstrate good performance on three tasks (image inpainting, super-resolution, and style conditioning).\n\n- The paper is generally well-written and is easy to follow.\n\nWeaknesses:\n\n- The assumption made in Eq. (11) that $J e$ is similar to $J^{-1} e$ needs further empirical/theoretical justification. Although the paper provides a short, intuitive explanation in line 133, I still have concerns regarding whether or how much the assumption holds. I think even empirical justifications (computing both terms and evaluating their difference) would be valuable.\n\n- I didn't really follow the transition from Eq. (10) to Eqs. (12) and (13). Specifically, I was confused by the introduction of the learning rate $\\lambda$ since it does not appear in Eq. (10). Could the authors explain more why this can be the case? One explanation could be that $e_t$ is defined as $J e \\lambda$. I also had trouble understanding why this is considered Newton's method, since Eq. (11) may be very different than $J^{-1} e$. Even if the analysis is sound, I believe a better explanation should be given in the paper to minimize confusion.\n\n- While the method claims to avoid back-propagation, it seems that Eq. (16) simply replaces auto-diff with a finite-difference approximation of step size $\\delta$. This can also be done with forward-mode auto-diff. It would be interesting to also compare their empirical performance.", "was_revised": true, "final_justification": "The rebuttal resolved most of my concerns. However, I do think the authors should give a better explanation on the assumption of $J \\approx J^{-1}$ in the next version.", "rebuttal": "We thank the reviewer for their thorough comments. We respond to the questions below:\n\n**The assumption made in Eq. (11) that $Je$ is similar to $J^{-1}e$ needs further empirical/theoretical justification**:\n\nWe understand that the intuition provided in line 133 may not be satisfying. The main difficulty of empirically demonstrating the similarities between $J$ and $J^{-1}$ lies in computing the Jacobian $J$ itself. Computing the Jacobian locally for one input $x_t$ could require ~16000 autodifferentiation passes, one for every row of the Jacobian. Understandably, for a model at the scale of Stable Diffusion (800M parameters), running an experiment where we compute the Jacobian at multiple input points is infeasible within the timeframe of this rebuttal.\n\nAs an alternative, we discuss a smaller-scale experiment with a small diffusion model (25M parameters) trained on the MNIST dataset. We padded the MNIST digit images with zeros to 32x32 pixel images, making the input and output of the model 1024-dimensional. Here, $J \\in \\mathbb{R}^{1024\\times1024}$ can be readily computed with automatic differentiation (~10s on our machine), allowing us to also compute $J^{-1}$ assuming $J$ is not singular.\n\nMeasuring the similarity between the two matrices is challenging; we find that they differ significantly in scale, with $J^{-1}$ having much larger values than $J$, which makes measuring some norm of their difference (e.g. Frobenius) unreliable. What we are interested in is whether the two matrices exhibit similar structures, i.e., given an input, whether they transform it in similar ways. To measure this, we can compute the eigendecomposition of the two matrices and compare the top-K eigenvectors. If the transformation defined by $J$ is similar to $J^{-1}$, we expect the most important eigenvectors to be similar, which we can easily measure with cosine similarity.\n\nWe perform the following experiment: Given a noisy image $x_t$ at timestep $t$, we compute the exact Jacobian $J$ using automatic differentiation (`torch.autograd.functional.jacobian`),  its inverse $J^{-1}$, and the eigenvectors and eigenvalues of both matrices. We then select the top 20 eigenvectors, excluding their imaginary parts, and ordering them by the magnitude of the corresponding eigenvalue. To account for misalignment between the eigenvalues, we pair each eigenvector of $J$ with its closest eigenvector from $J^{-1}$ and measure the average cosine similarity across all eigenvector pairs.\n\nWe run this experiment for different timesteps $t$, sampling 10 random images at each timestep from the MNIST dataset to compute $J$ and $J^{-1}$. As a baseline, we choose random $J$ and $J^{-1}$ matrices computed at the same timestep and repeat the experiment to show that the similarities measured are not spurious. \n\n| Timestep $t$ | 950 | 850 | 750 | 650 | 550 | 450 | 350 | 250 | 150 | 50 |\n| - | - | - | - | - | - | - | - | - | - | - |\n| Similarity for the same $x_t$ | 0.68 | 0.72 | 0.62 | 0.72 | 0.71 | 0.72 | 0.72 | 0.63 | 0.60 | 0.58\n| Similarity for random $x_t$ | 0.11 | 0.11 | 0.16 | 0.29 | 0.29 | 0.19 | 0.16 | 0.15 | 0.14 | 0.12 | \n\nBy showing the high similarity between the top 20 eigenvectors of the two matrices, we empirically demonstrate the intuition of line 133; the two transformations $J$ and $J^{-1}$ can be used interchangeably to compute the gradient direction for the constraint. A particularly surprising finding is that in some cases, we found almost the same eigenvector (cosine similarity $\\approx 1$) from the two matrix decompositions.\n\nAlthough we demonstrate this in a smaller-scale experiment, we believe it supports our argument in the paper for using an inexact Newton method that approximates the steps of $J^{-1}e$ with $Je$. We will include the above experiment and discussions in the paper to strengthen the intuitive explanation given.\n\n**Eqs. (10)-(13) and introduction of the learning rate**:\n\nEq. (10) tells us that we must compute an approximate solution $e^\\*_t$ that strictly reduces the residual error $r$ if we want the method to converge. The solution we choose in this paper is $e_t^* = \\lambda Je$. We agree that it is better to include the learning rate $\\lambda$ in Eq. (11) to avoid confusion.\n\nRegarding the role of the learning rate, the sole reason we introduce it is to ensure the convergence of the proposed update. Although in our experiments we tuned the learning rate empirically, in Appendix A we provide an in-depth discussion of how $\\lambda$ is related to the spectra for $J$ and affects the convergence. We will point readers to that in the main paper.\n\n**Why is this considered Newton's method?**:\n\nNewton's method involves solving for the inverse of the Jacobian $J^{-1}$ and performing the update $J^{-1}e$. In cases where computing the inverse is infeasible, we can approximate Newton's method with updates $Me$, where $M$ is chosen such that it is as close (in structure) to $J^{-1}$ as possible, but cheap to compute. These methods are called *Inexact* Newton methods and they have been studied in numerical optimization before (see Inexact Newton Methods [5]).\n\nOur algorithm utilizes an Inexact Newton method that approximates $J^{-1}e$ with $Je$. The reason this approximation works well is specific to denoising diffusion models and is discussed above.\n\n\n**Finite-difference approximation and exact gradient computation**: \n\nTo compute the $Je$, we use the finite difference approximation $Je \\approx (f(x + \\delta e) - f(x))/\\delta$. The automatic differentiation libraries implement the backward gradient computation, which gives the gradient descent direction $J^Te$. This is what `e.backward()`  computes in PyTorch.\n\nSome libraries also offer forward-mode auto-differentiation, which computes the Jacobian-vector product $Je$. However, forward differentiation is not always implemented or optimized as well as the backward propagation of gradients. In the case of PyTorch, which is what we use for running Stable Diffusion, forward mode differentiation is not directly implemented for many of the custom Stable Diffusion layers. \n\nTo perform a comparison between our finite difference approximation and the exact forward computation, we resorted to the double backward trick, which computes the forward-mode gradient with two backward calls. This is, of course, slower and memory-intensive, but we use it as a baseline to verify the finite-difference approximation employed in the paper. For completeness, we also **ablated the choice of $\\delta$**\n\nWe repeated the box inpainting task of Table 2 and present the results below:\n\n| Method | PSNR ↑ | LPIPS ↓ | FID ↓ | Time |\n| - | - | - | - | - |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.0005$ ) | 17.47 | 0.37 | 69.02 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.005$ ) | 18.30 | 0.30 | 42.01 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.05$ ) | 18.32 | 0.31 | 42.44 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, $\\delta = 0.5$ ) | 15.80 | 0.34 | 61.40 | 15s |\nOurs ($K=5$, $\\lambda = 0.5$, exact forward diff ) | 18.27 | 0.31 | 44.90 | 76s |\n\n $K$ number of steps, $\\lambda$ learning rate, $\\delta$ finite difference step size\n\nThe finite-difference approximation is (i) robust to the choice of $\\delta$, and only fails when using too small (0.0005) and too large (0.5) $\\delta$ values, and (ii) performs as well as the exact forward mode auto-differentiation. \n\n**Comparing the performance against the baseline with the same computation budget**:\n\nIn Appendix B.6, we briefly discuss what happens when the number of constraint optimization steps per denoising diffusion step is increased from 5 to 20. What we observed is that once the constraint optimization converges to a local optimum, the additional steps do not improve the final generated image and may also, in some cases, introduce unwanted artifacts. This is in line with the observations made by previous works (DPS [3]).\n\nWe believe that the empirical values for the number of steps $K$ and learning rate $\\lambda$ we chose in the paper are adequate for the optimization to converge sufficiently, and thus increasing the computation would not impact the result significantly. Since the results we obtained were already better than the baselines (see our response to Reviewer uiuT for even better super-res results), and our main argument is that training-free inference can be practical (<1min), we did not experiment much with increasing inference time."}, {"review_id": "i4MuqZvUB2", "reviewer": "Reviewer_sna1", "summary": "This paper focuses on constrained sampling in diffusion models (e.g., inpainting and super-resolution) and tackles heavy and high-overhead backpropagation in such tasks. Specifically, through rigorous derivation, the authors propose an efficient, high-quality generation paradigm that can operate under arbitrary constraints, which can be regarded as an approximation of Newton's optimization method. Experimental results demonstrate that the proposed approach is effective for inpainting, super-resolution, and style-guided constraint tasks.", "questions": "Please refer to the 'Weaknesses' section for details.\n\nMy primary unresolved concerns focus on the following aspects:\n\n1. **Writing and Presentations**;\n2. **Limited Evaluation Tasks and Necessity**;\n3. **Limited Quantitative and Qualitative Performance Gains**.\n\nOverall, while I would consider raising my rating if these concerns are adequately addressed, I think this paper still has significant room for improvement. Although I am not a leading expert in this specific field, based on my knowledge of ​generative models​ and the ​overall quality of this work, I think this paper is ​not yet ready for publication​.", "limitations": "Yes, the authors attempt to address the existing limitations.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n1. **Rigorous Theoretical Derivation**: The paper provides a thorough theoretical analysis and derivation of the proposed method, establishing a solid mathematical foundation.\n2. **Computational Efficiency**: Quantitative results in inpainting and super-resolution tasks demonstrate significant efficiency improvements over prior methods, reducing runtime from 6~8 minutes (previous works) to 15 seconds~1 minute.\n3. **Performance**: Both quantitative metrics and qualitative results on inpainting and style-guided generation tasks illustrate that the proposed method achieves consistent performance improvements over baseline methods.\n\n**Weaknesses**:\n\n1. **Writing**: While the paper provides meticulous theoretical derivations, the writing style is overly dense and tedious, making it challenging to read. The inclusion of nearly 20 mathematical formulas and pseudocode requires excessive time and effort from readers to comprehend. I recommend that the authors incorporate some highlighted, intuitive summaries to better guide readers through the methodological details and key insights.\n2. **Presentation**: Building on the previous point, the manuscript would benefit from adding a framework figure to intuitively illustrate the proposed method, which would help readers quickly grasp the innovation of the approach while making the content more accessible.\n3. **Limited Evaluation Tasks**: The paper claims the proposed method is applicable to arbitrary constraints, but the evaluation is restricted to only three tasks (inpainting, super-resolution, and style-guided generation). This narrow scope fails to adequately support the claimed generalizability.\n4. **Performance Trade-offs**: As shown in Table 1, while the proposed method achieves remarkable efficiency improvements, it demonstrates advantages only in inpainting FID and super-resolution PSNR metrics. For other metrics, the method falls behind baselines.\n5. **Limited Necessity and Performance Gains**: Table 2 reveals that the proposed method underperforms SD-Inpaint in both generation quality and inference efficiency. Although SD-Inpaint requires extra training, this substantial performance gap raises concerns about whether this training-free approach is truly necessary.\n6. **Visualizations**: Several concerning artifacts appear in the visual results: In Figure 3 (second row), the proposed method generates images with moiré-like strange textures between the sky and sea surface, which is not found in baseline methods. Moreover, Figure 5 shows that the proposed method produces less natural results compared to baselines, particularly in the first column (Ours), where the generated images appear unnatural with peculiar contours.", "was_revised": true, "final_justification": "I have carefully read both the other reviewers' comments and the authors' replies. While I'm not an expert in this particular field, I acknowledge that most of my concerns have been addressed. Thus, I have adjusted my rating accordingly.\n\nHowever, considering the paper's overall quality and contribution, I would suggest the authors carefully address the remaining issues raised by other reviewers, with particular attention to: (i) comprehensive experimental validation, and (ii) clearer presentation or writing logics.\n\nOverall, I think this work currently meets the basic standards for NeurIPS, though it still falls short of being 'exceptional' or 'groundbreaking'.\"\n\nI recommend placing more emphasis on the feedback from reviewers who specialize in this particular field.", "rebuttal": "We thank the reviewer for their time and constructive feedback. We respond to the weaknesses raised and questions below:\n\n**Limited evaluation tasks**:\n\nOur evaluation is performed with both linear (free-form inpainting, x8 super-resolution, box inpainting) and non-linear (style guidance) constraints. This is consistent with previous papers on training-free diffusion sampling algorithms, some of which are often limited to just linear constraints (e.g. PSLD [25], P2L [4]). Additionally,  in Appendix B.1, we included another non-linear task, *mask-guided generation*, which we omitted from the main text due to space constraints. There, we utilize a separately-trained face segmentation network to guide image generation with a target face mask. Any readers interested in mask-guided generation were pointed to it in line 60. Overall, we demonstrated our algorithm on a variety of tasks, showcasing its widespread applicability and consistent advantages over existing baselines.\n\n**Performance trade-offs**:\n\nWe argue that we can perform as good as or better than existing training-free sampling algorithms while only requiring a fraction of the inference time. We would like to emphasize that there is **no notable performance trade-off** when using our algorithm, as we discuss below.\n\n- Regarding Table 1, although our algorithm performs considerably worse than P2L in 8x super-resolution, we highlight two key disadvantages of P2L.\n  1. P2L requires 30 minutes to run on a single image, whereas our method only takes one minute.\n  2. P2L uses a VLM (PaLI) to caption the low-resolution images before the diffusion, which introduces additional steps to the inference.\n\n  As mentioned in lines 207-208, we believe that the main advantage of P2L in super-resolution comes from introducing text conditioning. All previous algorithms relied solely on the unconditional model to perform super-resolution, which may not be optimal when the model is required to synthesize specific high-resolution textures in the images.\n\n  To verify this assumption, we ran the x8 super-resolution with text prompts generated from a Qwen-2.5 VLM [Qwen]. We provided the downsampled images and asked the VLM to generate a text caption for each, which we then used as conditioning when running our algorithm. The results below indicate that we can close the performance gap and even improve upon the evaluation metrics when introducing text captions to the super-resolution process. Therefore, there is no trade-off in super-resolution performance when using our algorithm if we also introduce a VLM to caption the low-res images.\n\n  | Method | PSNR ↑ | LPIPS ↓ | FID ↓ |\n  | - | - | - | - |\n  | P2L (with VLM captions) | 23.38 | 0.386 | 51.81 |\n  | Ours | 24.26 | 0.471 | 60.99 |\n  | Ours + VLM captions | 24.95 | 0.405 | 44.74 |\n\n- In Table 2, which presents a more difficult task by inpainting 25% of an image, our method significantly outperforms all baselines and is only outmatched by the fully fine-tuned SD-Inpaint model, which required 100k additional steps of training. Similarly, in Table 3, our method consistently achieves lower style scores compared to the baselines.\n\nTo summarize, our method is substantially faster than all baselines and outperforms them on the more challenging tasks. To showcase this better, we will add the text-guided super-resolution result and the relevant discussion to the paper to complement Table 1.\n\n**Limited necessity of training-free sampling**:\n\nWe added the SD-Inpaint result in Table 2 as a theoretical upper limit of training-free approaches. We expected that training-free methods would produce worse results than the fine-tuned model, which required considerable additional compute to train. However, this comparison does not take away from the necessity for developing efficient and accurate training-free sampling algorithms:\n\n1. Training-free sampling algorithms can help understand the inner workings of these massive generative models. In the box inpainting task, we show that, using our method, Stable Diffusion accurately continues the textures from the given parts of an image in the inpainted regions, retaining the overall image structure (Figure 4). These findings can provide valuable insights into how these models understand and process images.\n\n2. For the linear tasks considered in this (and previous) training-free sampling papers, data can be easily obtained. However, there are cases where there is not enough data to fine-tune a model for conditional sampling. Recent works have applied the training-free sampling algorithms we compare to in our paper, in settings where no trained conditional model exists:\n   - [Reb1] Training-free image generation with image layouts and motion vectors as conditions.\n   - [Reb2] Training-free motion generation with text prompts, keyframes, and obstacles in 3D space as conditions.\n   - [Reb3] Training free molecule generation with geometric constraints.\n\n  Inpainting and super-resolution provide a good testbed for benchmarking training-free sampling algorithms. We show that our algorithm improves upon the performance and efficiency of existing training-free sampling algorithms, and future works can utilize our method as a drop-in replacement for the training-free conditioning guidance.\n\n**Visualizations**:\n\nThe artifact mentioned in Figure 3 is a watermark texture hallucinated by the model. Stable Diffusion was trained on web images, which were often watermarked, and can thus generate such patterns. We would like to point out that the hallucinated watermark is adhering fully to the imposed low-resolution image constraint. The mean squared error between the downsampled watermark image and the constraint is around 0.0008, which is also the mean squared error we get when we compare the constraint to another generated lighthouse image that has no hallucinated watermark. We agree that this image may cause unnecessary confusion to the reader and will replace it with another sample that has no watermark hallucination. We will move this example to Appendix B.7 More Qualitative Results.\n\nRegarding Figure 5, the baselines seem to copy the color palette, but do not exhibit the same brush strokes and textures visible in the reference style image. For instance, the tree texture from Van Gogh's painting (bottom row) has been correctly replicated in the cat's body in our generated sample, but is nowhere visible in the baselines. The baseline methods seem to only synthesize a cat fur texture with colors from the reference style. \n\nThis qualitative observation translates to the results of Table 3, where the proposed algorithm achieves a lower style score (lower is better), representing the style of the reference image more accurately. If imposing the constraint too strictly generates undesired images, a user can easily relax the constraint by reducing the learning rate $\\lambda$ or optimization steps $K$; it is an advantage that our algorithm provides the option to enforce the constraint as strictly as necessary.\n\n\n**Incorporate highlighted, intuitive summaries to better guide readers through the methodological details and key insights**:\n\nThe main intuition behind our method is that in diffusion models, we can approximate the Newton optimization step $J^{-1}e$ with several smaller steps in the direction of $\\lambda Je$, which is cheap to compute. This approximation is possible in denoising diffusion because $J$ and $J^{-1}$ exhibit similar structures, as discussed in lines 133-134.\n\nTo better highlight this intuition, as also suggested by Reviewer NFw9, we performed a smaller-scale experiment, where it is possible to exactly compute the Jacobian and compare $J$ and $J^{-1}$. For a diffusion model trained on MNIST, we showcased how the two matrices encode similar structures by comparing the similarity of their top 20 eigenvectors across different timesteps. Please refer to our response to Reviewer NFw9 for further details on this experiment.\n\nWe will include this toy experiment in the main paper, highlighting how we exploit the similar structures of $J$ and $J^{-1}$ to formulate our method. Moving beyond a toy dataset, e.g. a Stable Diffusion-scale model, is computationally infeasible, as a single Jacobian computation would require ~16000 backward passes through the model. Nevertheless, we trust that this toy example can help readers better understand our approach.\n\n**Adding a framework figure**:\n\nInstead of a framework figure, in which it would be difficult to visualize the difference between the various gradient updates in high-dimensional space, we will add examples of the eigenvectors of $J$ and $J^{-1}$ from the toy experiment above. The eigenvectors of $J^{-1}$ are images that showcase how the model encodes correlations between image pixels, i.e., how the model input should change for a constraint applied to its output. By showing that our proposed $Je$ update captures similar correlations to what the optimal inverse update $J^{-1}e$ would, we provide a visual explanation of the intuition behind our method.\n\n\nAdditional references:\n\n[Qwen] Bai, Shuai, et al. \"Qwen2. 5-VL technical report.\" arXiv preprint 2025\n\n[Reb1] Wang, Zixuan, et al. \"Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis.\" CVPR 2025\n\n[Reb2] Karunratanakul, Korrawe, et al. \"Guided motion diffusion for controllable human motion synthesis.\" ICCV 2023\n\n[Reb3] Ayadi, Sirine, et al. \"Unified guidance for geometry-conditioned molecular generation.\" NeurIPS 2024"}]

Author rebuttal:

Meta-review: This paper proposes an algorithm for performing diffusion sampling under target constraints. It's often time-consuming to finetune the pretrained diffusion model on the new constraints and the cost can be reduced through constrained sampling. The authors propose to use second-order method under linear constraints to perform sampling for a diffusion model and approximate the jacobian inverse with jacobian. The proposed algorithm runs much more efficient compared to baselines and achieves on-par results. Reviewers are concerned that the performance is sacrificed (reviewer WTsj), not state-of-the-art (reviewer uiuT), the approximation of jacobian (reviewer, NFw9), and lack of novelties. The discussion has cleared most of the concerns among reviewers.  Reviewer WTsj is concerned on trading performance with efficiency. The algorithm demonstrates a strong gain on efficiency while the performance fluctuation is within some reasonable range. I'd suggest this paper as a candidate for acceptance.
Decision: Accept (poster)