Diffusing Differentiable Representations
Authors: Yash Savani
Marc Anton Finzi
J Zico Kolter
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion models
Differential Geometry
Implicit Neural Representations
NeRF
Siren
Primary area: diffusion_based_models
TL;DR: We sample differentiable representations by solving the pulled back reverse diffusion process in parameter space.

Abstract: We introduce a novel, training-free method for sampling *differentiable representations* (diffreps) using pretrained diffusion models. Rather than merely mode-seeking, our method achieves sampling by "pulling back" the dynamics of the reverse-time process—from the image space to the diffrep parameter space—and updating the parameters according to this pulled-back process. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects. Our method yields diffreps with substantially **improved quality and diversity** for images, panoramas, and 3D NeRFs compared to existing techniques. Our approach is a general-purpose method for sampling diffreps, expanding the scope of problems that diffusion models can tackle.

Diffusing Differentiable Representations
Yash Savani
Marc Finzi Carnegie Mellon University mf ffiinii Candrew cmu edu
J. Zico Kolter
Carnegie Mellon University ysavani Candrew. cmu. edu
Carnegie Mellon University zkolter@andrew cmu. edu
Abstract
We introduce a novel, training-free method for sampling differentiable represen- tations (diffreps) using pretrained diffusion models. Rather than merely mode- seeking, our method achieves sampling by "pulling back" the dynamics of the reverse-time process-from the image space the diffrep parameter space-amc and updating the parameters according to this pulled-back process. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects. Our method yields diffreps with substantially improved quality and diversity for images, panoramas, and 3D NeRFs compared to existing tech- niques. Our approach is ceneral-purpose method for sampling diffreps, expanding the scope of problems that diffusion models can tackle.
1 Introduction
Diffusion models have emerged as a powerful tool for generative modeling [Ho et al. 2020 Song
et al. 2022 2021 Karras et al. 2022 Rombach et al. 2022]. , and subsequent work has extended
these models to generate complex objects-suuce as 3D assets, tiled images, and more. Although
such approaches typically require training or least fine-tuning of the diffusion models for these
new modalities [Wang et al. 2023 Luo et al.) [023]. two notable exceptions, Wang et al. [202]
and Pooole et al. [2022]. have developed methods for training-free production of 3D objects by directly using an image-based diffusion model. Both methods work by optimizing a differentiable representation (diffrep)-in this case, a Neural Radiance Field (NeRF) [Mildenhall al. 2020) to
produce rendered views consistent with the output of the image- based diffusion model. Unfortunately. the nature of both these methods is that they optimize the diffrep to produce the "most likely" representation consistent with the images; that is, they perform mode-seeking rather than actually sampling from the diffusion model. This results in overly smoothed outputs that lack detail and do not reflect the underlying distribution of the diffusion model.
In this paper, we present novel method for sampling directly in the diffrep space using pretrained diffusion models. The method is training-free. can handle arbitrary diffreps, and performs true sampling according to the underlying diffusion model rather than merely mode-seeking. The key idea of our approach is to rewrite the reverse diffusion process itself in the diffrep parameter space. This achieved by "pulling back" the dynamics of the reverse time process from the image space the parameter space and solving (small) optimization problem, implied by the malled-back, dynamics, over the parameters for each diffusion step. To further encourage solver convergence, we identify constraints that the diffrep induces on the samples and use these constraints to guide the reverse process to generate samples from the high-density regions of the diffusion model while satisfying the constraints along the sampling trajectory.
Our experiments use a pretrained image-based diffusion model (Stable Diffusion 1.5 [Rombach et al.) 2022) to generate samples from various classes of diffreps. For example, by sampling SIREN representations [Vincent 2011 with wrap-around boundary constraints, we can sample total 360- degree panorama scenes, and by sampling NeRF representations MMlleenhall et al. 2020), we can
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generate 3D models of many objects. In both settings well as in baseline comparisons on simple SIREN-based (non-panorama) image generation our approach substantially outperforms previous methods such as Score Jacobian Chaining (SIC) (Wang et al., 2022]. Though the problem setting considerably more difficult without fine-tuning or retraining, sampling diffreps using pretrained diffusion models with our method substantially improves and extends the state-of-the-art in training-free generation methods.
2 Related work and preliminaries
2.1 Differentiable representations (diffreps)
Diffreps are powerful tool for representing complex scenes or manifolds using parameterized differentiable functions to map coordinates of the scene into a feature space that encodes the properties of the scene at those coordinates. Popular instantiations of diffreps include: SIRENs (Sitzmann et al.) 2020J. which implicitly model an image using an MLP with sinusoidal activations to map 2D (x. y) pixel coordinates into RGB colors, and NeRFs [Mildenhall et al. 2020 which implicitly model a 3D scene using an MLP that transforms 3D (I. y; 2) voxel coordinates with view directions (t) 16) into RGBar values. We can render images from differrntt views of the scene by numerically integrating the NeRF outputs along the unprojected rays from the camera.
Faster or alternative diffreps also exist for 3D scenes- -ucch as the InstantNGP [uüller et al. 2022 model or Gaussian splats (Kerbl et al. 2023] althoogh in this work we focus on the basic NeRF architecture. Many other kinds of visual assets can also be formulated as diffreps. For example, we can use diffreps model panoramas, spherical images, texture maps for 3D meshes, compositions of multiple images, scenes from movie, and even the output of kinematic and fluid simulations.
Many interesting diffreps can be used render not just one image from the scene but multiple coupled images or even distribution over images. Given a diffrep, parameterized by 0 E O for a scene-such as SIREN panorama or a NeRF- we can render an image of the scene from view π E II using a differentiable render function f(0, π) = image E X. To accommodate the multiview setting in our discussion, we consider the "curried" Haskell form of the render function f : θ + (II X), where f()) II + X' is map itself from view π E II to ( f(0, T) = image E X.
In the of NeRFs SIREN panoramas, the view π is continuous random variable drawn from a distribution that, with abuse of notation, we will also call II. To formalize this, let 7( S X be vector space of functions from II to X. With this definition,
can write the signature of f as f: @ + H. Although H is usually larger than X' (and can be even infinite in some cases), we can simplify our notation by equipping H with an inner product to make it a Hilbert space. This allows us to use familiar matrix notation with H and hide the view dependence of f(0) in H.
To complete this formalization, we lift the inner product on X' to define inner product H:
Yh.g g H.: gh. == E,-n (h(T)) g(+)) Em-u(h(m) (+())) g(=)].
This formulation allows to handle the multiview nature of NeRFs and SIREN panoramas in mathematically rigorous way while preserving the convenient notation.
The partial application of f returns entire view dew-deenndent function f(0) H. Given set of images (Z))e(NN) of the scene from different views (π.)iE[N). the standard diffrep task is to find a 0 E ④ that minimizes C(0) = Eie[n) (0),) - xill. Because the diffreps and f are both differentiable, this can accomplished using first-order solvers, such as gradient descent or Adam.
For pedagogical convenience, we identify H. with X' in the following sections. This identification simplifies the presentation, provides more interpretable and intuitive perspective on our method, and is precise when II is a singleton. Because H is Hilbert space, the main points of our arguments still hold when we consider a larger II. We describe the specific changes needed to adapt our method to the general multiview setting when H + X, in susseecction 3.2
2.2 Diffusion models
Diffusion models implicitly define a probability distribution via "reversing" forward noising process. While many different presentations of image diffusion models exist in the literature (c.g., DDPM

---

[o et al. 2020], DDIM (Song et al. 2022], Score-Based Generative Modeling through SDEs [Song
al. 2021], EDM (arrass et al. 2022)), they are all equivalent for our intended purposes.
Given a noise schedule (t) for E [o, T and a score function V log pr (x(()) for the distribution Pt over noisy images s=t) at time i, and then following the reverse time probability flow ODE (PF-ODE) given in Karras et al. [2022] to
sample from po by first initializing x:() ~ (O, o² (T) (T)
transform the easy-to-sample x(T) into a sample from po:
dx dt = (()) (oge)((())ooop((((())
The PF-ODE is constructed so that the perturbation kernel (conditional distribution) is given by Pos (o)(=(t))=(0)) = (()) (t).). Using the reparameterization trick, we can write this as a<()) = =0) + (1)), where e N/
We approximate the score function with (learned) noise predictor t)) ~ z(d) ((a-)(((0] using
Vlogp( B[e(O)]a(t) - a(t)) a(t)
((x()).))
the Tweedie formula RRobbins 1956
Efron 2011
x²(1)
(()
2.3
Training differentiable representations using diff'usio priors
2.3.1 Training-free methods
Poole et al. [002 laid the foundation, demonstrating that generating 3D assets from purely 2D image-trained diffusion models was even possible. In DreamFusion (SDS), they perform gradient ascent using [u,,(w(t))]' ((())) + (tt),,t) - =)]], derived from the denoising objective, where is the Jacobian of the differentiable render functioo f and é is the learned noise predictor.
Independently, Wang al. [0022] introduced Score Jacobian Chaining (SJC), which performs gradient ascent using V Vlog p(0)= == [J V log (f + (t)c)) with a custom sampling schedule for the ts. The custom schedule can be interpreted as gradient annealing align the implicit (t) of the diffrep with the used to evaluate the score function.
Comparison of methods Using the Tweedie formula, we can rewrite the SDS objective as V log + (t)e) This expression identical the llgg((0) term from SJC if we let wt) (t) for all t. Both methods follow this gradient to convergence, which leads to a critical point-a a local maximum or mode-in in the log p(0) landscape. To approximate the objective, both approaches
Er,e
w (t))
Monte Carlo sampling.
Both approaches optimize the objective using gradient ascent (GA). While GA on the transformed score resembles solving the PT-ODE-particularyy in their discretized forms-the two procedures serve fundamentally different purposes. GA cares only about finding a local maximum in the log p()) landscape, regardless of the trajectory taken to get there. In contrast, the PF-ODE follows specific path to ensure that generated samples are typical of the distribution. For a discussion of the limitations of GA and why it does not produce representative samples from the distribution, see Appendix A
Recent developments Several subsequent works have built upon the SDS and SJC methods by incorporating additional inputs, fine-tuning, and regularization to enhance the quality of the generated 3D assets. Zero-1-to-3 [Liu et al.] expands on SJC by fine-tuning the score function to leverage single real input image and additional view information. Magic 123 [Qian et al.] further builds on this by incorporating additional priors derived from 3D data. Fantasia31) [Chen al.) separates geometry modeling and appearance into distinct components, using SDS to update both. HiFA [Zhu et a introduces modified schedule and applies regularization to improve the SDS process. Finally, LatentNeRF [Metzer et al.) utilizes SDS but parameterizes the object in the latent space of stable diffusion autoencoder, rendering into latent dimensions rather than RGB values. While these methods rely on the SDS/SIC framework for mode-seeking, our work takes a wholly different approach, focusing on developing a more faithful sampling procedure to replace SDS/SJC for both 3D generation and broader differentiable function sampling.
Note that [5|] = 0, so the additional -e term in the SDS objective behaves as an unbiased variance reduction term for the score estimate since Cov(é, > O.
3

---

2.3.2 Pretrained and fine-tuning methods
Unlike SDS and SJC-whic are entirely zero-shot and can be performed with frozen diffusion severaseveral other methods have been developed to achieve improved quality and diversity of
the generated diffreps, albeit at the cost of additional fine-tuning. While some of these methods
require additional data [Zhang et al]]. ProlificDreameer (VSD) [Wang et al. [023] and Difflnstruct
[Luo et al., 2023) are two examples in which diffusion models can be fine-tuned or distilled using only synthetically generated data.
VSD specifically addresses the problem of generating 3D assets from a 2D diffusion model using particle-based variational inference to follow the Wasserstein gradient flow, solving the KL. objective from SDS in the (2) parameter distribution space. This approach produces high-quality, diverse samples even lower guidance levels.
In our work, we restrict ourselves to the training-free setting. This choice more readily enables application new modalities and is independent of the score function estimator's architecture.
2.4 Constrained sampling methods
Our method requires generating multiple consistent images of scene from different views. For instance, we generate various images of 3D scene from different camera locations and orientations to determine NeRF parameters. These images must be consistent to ensure coherent updates to the diffrep parameters. While the diffirep inherently enforces this consistency, we can improve our method's i convergence rate by encouraging the reverse process to satisfy consistency conditions using constrained sampling methods.
Several approaches enable constrained sampling from an unconditionally trained diffusion model. The naive approach of projecting onto constraints (Song et al., [202]] leads samples lacking global coherence. In contrast, Lugmayr et al. [2022] propose RePaint- simple method for inpainting that intermingles forward steps with reverse steps harmonize generated noisy samples with constraint information. For references to more general methods used to constrain the distribution by conditioning
the score function, see Appendix
Pulling back diffusion models to sample differentiable representations
The score model associated with the noise predictor implicitly defines a distribution the data space X, and the reverse time PF-ODE from Eq. I provides a means to sample from that distribution. In this section, we will show how use the score model and the PF-ODE sample the parameters of differentiable representation (diffrep) so that the rendered views resemble samples from the implicit distribution.
SDS and SJC derive their parameter updates using the pullback of the sample score function through the render map f- This pullback is obtained using the chain rule: d(log d(log de The pullback score is then used as for gradient ascent the parameter space. However, since p is a distribution- a simple scalar field the change of variables formula requires an additional log det J volume adjustment term for the correct pulled-back score function in the parameter space, where J is the Jacobian of f. A careful examination of this approach through the lens of differential geometry reveals even deeper issues.
d9
de
In differential geometry, it is a vector field (a section of the tangent bundle TX') differential 1-form (a section of the cotangent bundle T'X) that defines the integral curves (flows) on manifold. To derive the probability flow ODE in the parameter space, we must pull back the sample vector field dr 28
E T'' to the parameter vector field d0
di E TO using f. The pullback of the vector
field through / is given by (J* dr)le )lo (JJ di ((0), Thus, we derive in Appendix C the
de
pullback of the probability flow ODE is:
d0 di
(())(((())))))))))))))))) Vlogooo
We note that this is in contrast to the score chaining from Poole et al. [2022] Wang et al. 2022 The confusion stems from comparing the different types of puak elements. While de in Eq.

---

θ
X
d = (J
de
da
dt
TO
TX
(JTJ) -
JJ
3
T* JT V log logp(r) llgp(#)
T*
Vlogp(0) J V
(a) (J" de (b) J Vlgg(()) (c) XJ V log p(æ)
Figure 1: (Left) Commutative diagram showing how the PF-ODE vector field gets pulled back through f. respecting the differential geometry. The process involves: ① converting dr to the cotangent vector field V p(r) (up to scaling terms) with the Euclidean metric 1, @ pulling back V log p(x) via the chain rule using the Jacobian J, and then 3 transforming the pulled back differential form score function into the corresponding vector field using the inverse of pulled back metric (J² J)-1. When used in PF-ODE, SJC and SDS take the bottom path with the chain rule, however they do not complete the path by neglecting the T - TO transformation. (Right) SIREN image renders generated using the PF-ODE schedule with the prompt "An astronaut riding a horse" using the: (a) complete pulled-back da vector field, (b) pulled-back covector field from SJC (omitting
step ()) J (xx), (c) Scaled pulled-back covector field from SJC 0.0001.
a vector field, the score function V log pe (x(()) a covector field differential form. Pulling back the score function as a differential form correctly yields J (f)) term used in SJC. The issue lies with the hidden (inverse) Euclidean metric within Eq. I which converts the differential-fone score function into the corresponding vector field.
In canonical coordinates, the Euclidean metric is the identity. Consequently, the components of the score function remain unchanged when transformed into vector field. Therefore we safely ignore the metric term in the PF-ODE formulation of Eq. 1 for the sample space X. However, this is not true for the diffrep parameter space +
To convert the pulled-back score function into the corresponding pulled-back vector field, we must use the pulled-back inverse Euclidean metric given by (J J)- This yields the pulled-back form of the PF-ODE in Eq. 2P Fig. illustrates this procedure via a commutative diagram and provides an example of what goes wrong if you use the incorrect pulled-back term, suggested by SDS and SJC. In the SDS and SJC approaches, the (JT J) term behaves like a PSD preconditioner for gradient ascent. It may accelerate the convergence rate to the mode but does not fundamentally change the solution. However, this term is critical for the pulled-back PF-ODE since it impacts the entire trajectory and the ultimate sample (see Appendix A for more discussion). For explanation of why the log det. J term is absent the mathematically correct update, see Appendixx C
A more intuitive way to understand why the correct update is given by Eq. rather than SDS and SJC's version to consider the case where the input and output dimensions are equal. From the chain de
di di where J is f's Jacobian. This leads de J--1d equivalent. For non non-invertible f. we can interpret the pullback as the solution to the least- least-squares minimization problem arg min s0 do de
rule, we have d8 de de Using the inverse function theorem, compute du
=
dz 20
J-1
de' For invertible f. this equation and Eq. are
d2
3.1 Efficient implementation
Separated noise
Following the pulled-back PF-ODE in Eq. 2 we can find 0, such that f())) represents a sample ær()) Porpooottttt (=(t)==()) = N (z()); (t)]), However, this approach has limitation: zxt) is noisy, and most diffrep architectures are optimized for typical, noise-free images. Consequently, J is likely ill-conditioned, leading to longer convergence times.
*One may be tempted to impose that the metric is Euclidean both in the X' space and the θ space, but these choices incompatible because the render transformation maps between the two spaces.
5

---

forward & backward
f(0))
2o(()
60%
a(1)
The parameters of the diffrep 0; θ (torus) are used to render the noiseless signal (0;) ot(t). which are then combined with the noise (t)e to generate the noisy sample z(t). We can pull back each step of the reverse diffusion process to update the parameters 0, + A0.
Figure 2:
To address this issue, we factor ætt) into noiseless signal and noise using the reparameterization of the perturbation kernel: æ(t) = (t) + (t))((t). By letting c(t) remain constant throughout the sampling trajectory and starting with (T) O, we can update To with deg da
dt di (t) This
decomposition allows f(,) to represent the noiseless (t) instead of xx)), substantially improving conditioning of J. Fig. illustrates how this separation works in practice.
The Jacobian J in Fq.. 2 represents the derivative of the image with respect to the diffrep parameters. For all but the smallest examples, explicitly forming J .J is computationally intractable. While iterative linear solvers together with modern automatic differentiation frameworks (c.g.. (Potapezynski al., 2024)) could solve Eq. 2] we found faster to compute the parameter update as the solution to a non- linerrnear optimization problem. This approach avoids the costly JVP (Jacobian vector product) required to multiply with J.
Efficient optimization
For each step with size At, we solve:
A
arg min (0) - 1(0 50) - (1)AI((((())) + (1)e, - -
59
As At. - 0, using the linearization f(0 A0) f(0) JA0 (11 this optimization objective approaches the linear least-squares solution. The optimal 49 converges to the exact de
77 from Eq. 2 In practice, we employ the Adam optimizer to solve [qq 3 for whatever discrete At is used the diffusion sampling process. This procedure avoids the explicit formation of the linear least-squares solution.
3.2 Coupled images, stochastic functions, and 3D rendering
Thus far, we have only considered the case where the output of the render function is single image X. However, our interest lies in render functions f that map parameters θ to an entire view-dependent image space H C I Using the inner product defined in subsection 2. we derive view-dependent pullback PF-ODE:
d0 = dt
dōo
1* dx o())e = -O) J=] BB JJ (1o)))()()))
((
<(T)
di
(4)

---

where the expectations are taken over the views ~ II. Here, V ( represents the view-specific score function. We use the original score function V log Pa (a(())), with additional view information in the prompt. <(T)) denotes the separately managed noise rendered from view л.
Examining the optimization form of this equation provides further insight. We interpret it as minimizing the norm in the function space (I] 1l = Es-Hrll I1²): This is directly analogous to Eq. but with a different inner product. Explicitly written:
A
arg min B [1f(0, m f(0 A0, x) - -
(5)
A
which we can empirically minimize using view samples II.
3.3 Consistency and implicit constraints
Our method successfully "pulls back" the PF-ODE de using the pull-back of the score function V logg Pt (x(()). However, our true goal is to pull back V log (x) E range( for all s I t. This ensures that can render the noiseless components To throughout the remaining reverse process. For example, when pulling back the PF-ODE for different views of a 3D scene, we want to ensure consistency across all the views.
at
When f sufficiently expressive and invertible, V (z E range(f)) & V (x)))()())), This would allow first to sample z0) using the PF-ODE from Eq. 1 and then invert to find 8(0). However, for most significant applications of our method. f is not invertible.
In noninvertible cases, consider the sampling trajectory x=t) in X' when 2o(t) & range f. particularly when no nearby sample has high probability range f. Each step of Fqq. follows the direction in θ that best approximates the direction of the score model in least-squares sense. When f is not invertible, J is not full rank, and the update to will not precisely match the trajectory in x. Consequently, I (0, A0) - z(t At)ll will be significant. The score model, unaware of this, will continue to point towards high-poe gigh-probability regions outside the range of f.
For example, consider an f that only allows low-frequency Fourier modes in generated samples. The unconstrained score model might favor images with high-frequency content (e.g., hair, explosions, detailed textures), resulting in blurry. unresolved images. However, suppose the score model was aware of this constraint against tigh-frequency details. In that case, it might guide samples toward more suitable content, such as landscapes, slow waves, or impressionist art dominated by expressible low-frequency components. This issue becomes more pronounced when sampling multiple coupled images through f, such as with panoramas and NeRFs, where various views must be consistent.
To address this challenge, we adapt the RePaint method [Lugmayr al. 2022 initially designed for inpainting, to guide our PF-ODE towards more "renderable" samples. RePaint utilizes the complete Langevin SDE diffusion process, interspersing forward and reverse steps in the sampling schedule. This approach allows the process to correct for inconsistencies during sampling. The forward steps harmonize samples at the current time step by carrying information from earlier steps that implicitly encode the constraint. RePaint requires a stochastic process for conditioning, so we employ the DDIM [Song et al., [022) sampling procedure with controllable noise at each step. We derive the forward updates of DDIM in Appendix D
3.4 Summary
Our method introduces significant advancements for sampling difffreps using pretrained diffusion models by performing actual sampling instead of merely mode-seeking, thus capturing the full diver- sity of nigh-dimensional distributions. We derive the correct pullback of the PF-ODE, incorporating the essential (J"J)" JJ term to ensure unbiased sampling. Efficiency enhanced through separated noise handling and faster suboptimization process, allowing practical application for complex representations. Additionally, our approach extends to coupled images and stochastic functions with implicit constraints, making it suitable for tasks like panorama generation and 3D rendering. comprehensive summary of the contributions this section and the pseudocode for the DDRep algorithm using DDIM sampling with RePaint is presented in Appendix E
7

---

urs
CFG 0.0
G.G3.0 3.0
CFG 10.0
CFG 30.0
CFG 100.0
0.25
5JC
SD ndt
0.20
0.35
KID
Ours
0.30
0.03
sjc
0.00
CFG Scale
Figure 3: The left figure contains sample renders using the prompt "A woman is standing crosswalk at a traffic intersection." from the reference SD (top), our method (middle), and SJC (bottom) over the CFG scales [0,3,10. 10,30,0000000000 300] from left right. The right plot is the KID metric (closer to 0 is better) measured on the SIRENS sampled from our method, the SD reference samples, and the SIRENS sampled using SJC.
50 RAI
O
SIG
Figure 4: Samples generated using SD ref (top), Our method (middle), and SJC (bottom) using the same prompt office cubicle with four different types computers" with eight different seeds.
4 Experiments
All our experiments used the Hugging Face implementation of Stable Diffusion 1.5 (SDv1.5) 1.5) from Rombach et al. [2022) with the default hyperparameters as the noise predictor model. We conducted our experiments on single NVIDIA A6000 GPU. We used the complete Langevin SDE given by
the DDIM (Song et al., 2022) procedure for all our experiments, with 0.75 the stochastic
interpolation byperparameter.
interspersed forward and reverse steps to harmonize the diffrep
constraints in the sampling procedure. For the suboptimization problem described in Bq. we used Adam optimizer for 200 steps.
4.1 Image SIRENs
We compare the results of our method with those of SJC [Wang et al. 2022 when used to produce SIRENs [Vincent, 2011 for images. While could directly fit the SIREN to the final image
(00) obtained by first solving the PF-ODE the image space-we we use this scenario to compare different training-fii frree generation methods quantitatively.
The SIREN we used was a 3-layer MLP mapping the 2D (x, [0, 1)s pixel coordinates to the R4 latent space of SDv1.5. The ML.P had 128-dimensional sinusoidal embedding layer, depth of 3, a width of 256, and used sin(-) as the activation function. render an image, we used grid of equally spaced pixel coordinates from [0, 0] to [1, 1] with shape of (64, 64, 2). The MLP
64

---

CFG Scale 10.0
No RePaint RePaint
Figure 5: Comparison of landscape panoramas sampled using method. The top panorama is sampled using the RePaint method, while the bottom sampled without RePaint. Both approaches use 460 function evaluations (NFEs) to ensure faimess and CFG scale of 10.0. The prompt for these panoramas was "Landscape picture mountain range in the background with an empty plain in the foreground 50mm f/1.8".
output was latent image with shape (64, 64, 4), which we decoded using the SDv1.5 1.5 VQ-VAE into an RGB image with shape (512, 512, 3). We calculated the final metrics on these rendered images.
We used the first 100 captions (ordered by id) from the MS-COCO 2017 validation dataset Lin al.] as prompts. We generated three sets of images: (1) reference images using SDv1.5, (2) images rendered from SIRENs sampled using our method, and (3) images rendered from SIRENs generated using SJC [Wang al., 2022]. We compared these sets against 100 baseline images generated using SDv1.5 with the same prompts but different seed. We repeated this experiment at five CFG scales (Ho and Salimans] (0, 3, 10, 30, 100). We used the KID metric from Binkowski al,, to compare the generated images, it converges within 100 image samples while maintaining good comparison ability. For runtime metrics, see Appendix F
The example images in Fig. 3 (left) demonstrate that the samples generated by our method are almost indistinguishable from the reference. The results in Fig. 3 (right) clearly show that our method produces images on par with SDv1.5 reference samples.
To further support this observation, measured the PSNR, SSIM, and LPIPS scores of our method's samples against the SDv1.5 reference images (Table 1). The results show that our method produces SIREN images nearly identical the reference set.
To demonstrate that our method achieves sampling instead mode-seeking, we plot eight samples using the same prompt for all models in Fig. with different seeds at CFG scale 10. The samples generated by SJC all look very similar, while those generated by our method and SD show significantly more diversity.
4.2 SIREN Panoramas
In this section, demonstrate how method generate SIREN panoramas using the same architecture as the SIREN from Subsection 4. with a modified render function. We begin by sampling view grid of pixel locations with the shape (8, 64, 64, 2) where the batch size is 8, the height and width are 64, and the last 2 dimensions are the (I. y) coordinates. The y values are equally spaced in the range [0.1], and æ values are equally spaced the range [r, 1/ar] mod 1.0 for r (0, 1). The mod 1.0 constraint ensures horizontal wrapping, making the panorama continuous over the entire 360 azimuth.
We render the view by sampling from the SIREN at the discrete pixel locations given by the view grid, then decode the latent output using the SDv1.5 VQ- VAE. For experiments, assumed that the images were taken by a camera lens with a 45% field view (equivalent to 50mm focal length), corresponding to aspect ratio of
8.
The SIREN panorama imposes implicit constraints to ensure consistency across overlapping views. Fig. 5 and Fig. 7 iillustrate the impact of the consistency conditioning described in subsection 3.3

---

Figure 6: NeRFs generated using method with the prompts (top) "a photo of delicious hamburger, centered, isolated, 4K.", (middle) "a DSLR photo of rose", and (bottom) "a DSLR photo of yellow duck".
Both RePaint and non-RePaint methods use the same number of function evaluations (460 NFEs) to ensure fair comparisons.
Using RePaint substantially improves the quality and consistency of the generated panorama. This improvement is even more pronounced at low CFG scales, where the diffusion model is encouraged to be more creative and is typically less likely to produce consistent images, as seen in Fig. For runtime metrics and additional panoramas generated using our approach, see Appendix
4.3
3D NeRFs
For our NeRF experiments, we employed the voxNeRF architecture used by SJC [Wang et al. 2022 chosen for its speed and efficiency. While we did not utilize any additional loss terms from their method, we found that incorporating the object-centric scene initialization from Wang et al. [2023) significantly improved our results. This initialization bootstrapped the diffusion process and led
to faster convergence. Specifically, we used the initialization function (nni(()) = 10
0.5
|x|
as
detailed Wang et [2023]. For more experimental details, see Appendix F
The NeRFs generated using our method, as illustrated in Fig. 6, demonstrate our approach's ' capability to produce detailed, high-quality 3D representations from text prompts. These results showease
the effectiveness of our method in generating complex 3D structures while maintaining consistency
across multiple views.
5 Conclusion
We have presented comprehensive, training- frining-free approach for pulling back the sampling process of a diffusion model through a generic differentiable function. By formalizing the problem, we have addressed two critical issues with prior approaches: enabling true sampling rather than just mode seeking (crucial at low guidance levels), and addressing a latent consistency constraint using RePaint to improve generation quality. For future work and limitations, see (epeenddi G
Our method opens up new possibilities for generating complex and constrained outputs using pretrained diffusion models. We hope that future research will build on these insights to further expand the capabilities and applications of diffusion models in areas such as 3D content generation, game design, and beyond.
10

---

References
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. http:://arxx..org//aaaaaaa
11239 arXiv:2006. 1239
11239 [cs, stat].
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models, October 2022. URLbttp://axxiy org. abs/000/./000/ 02502 arXiv:2010.22502 [cs].
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- Based Generative Modeling through Stochastic Differential Equations, February 2021. Utt.ttpp://axx http:/awww....ooaaaaaaaaa abs/ 2011 13456 arXiv:2011.14456 [cs, stat].
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion-Basse Generative Models, October 2022. URL http:///axiv..rr/abbb 00364, arXiv:2266.03364 [cs, stat].
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://arxiv 12Xi::2112.10755 [cs].
org/abs/2112. 11722
Zhengyi Wang, Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- Dreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, May 2023. URL http: atw.v org/ httg/abs/23055 2205. 16213 MtXiv:2305.16213 [cs].
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models, May 2023. URL http:/tt://aaxxi org/ abs/ /2005. 18455, arXiv:2305. 18455 [es].
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation, December 2022. URL
htttp//arxiv.orge/ 221/ 00774 arXiv:2212.0077: [cs].
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using
2D Diffusion, September 2022. htLphttp://arxxt hrng/abs/22009 14988 arXiv:2209.
[cs, stat].
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, August 2020.
URL http://axxxx wttp:/or//////o////o/ 08934 arXiv:2003.0892. [cs].
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Com- putation, 2367):6661 July 2011. ISSN 0899-7667. doi: 10.1162/NECO__
a 00142. URL
bttts://ddi..orge 11621116/NNN NECO a_ 00142
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit Neural Representations with Periodic Activation Functions, June 2020. URL [cs, eess].
http://arxiv.. org/ wttp://org/abba/aaaeaas abtpa/ab.///////////// 09661 arXiv:2006.0966 09661
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics, 41(4):1- 15, July 2022. ISSN 0730-0301, 1557-7368. doi: 10. 10.114////22233331127, URL http: arXiv:2201.05993 [cs].
//arxiv. org/ abs 05989
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering, August 2023. URL httts://aaxxi org/
abs abbs/2308.04079v1
Herbert Robbins. An Empirical Bayes Approach to Statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Vol- ume 1: Contributions to the Theory of Statistics, volume 3.1. pages 157-164. Uni- versity of California Press, January 1956. URL httpss//projeeteecldd..rrr
Proceedings- chapter http:://wwa-nppiiiaaaaaaaaaaaaee http://hwww....aaaooooooooooontn

---

Bradley Efron. Tweedie's Formula and Selection Bias. Journal ofe American Statistical Association
106(496):6002-611,, 2011. ISSN 0162- 1459. URL htttps::://ww ncbi nlm. www..ii................... nww.ggo/oo............
articles/PMC332556//
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakoy, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot One Image to 3D Object. URLhtpp://rxxit http://wwwwwrraaaa////aa/ 11328
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. URL
https://arxiv wwwpiii.org//bb..
17843v2
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D:: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. /ttp://////ar htt//bbb/2303
13873
Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance. URL http:///: org/ abs// 0 18766
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent- for Shape-Guided Generation of 3D Shapes and Textures. URL https::/aaxiii..rgeeee.. hbwpgg/bb//2/// 07600v1
Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, and Jianfei Cai. Taming Stable Diffusion for Text 360()eeg) Panorama Image Generation. URL
http://arxii: /ttp///axvv..v.o wwtpsiivvorrvvieooo org/ heng/abs/24000 2404. 07949
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models, August 2022. URLhttp/ //arxiv.orgg absb/2011. 09865 arXiv:2201.0986 [cs].
Andres Potapezynski, Mare Finzi, Geoff Pleiss, and Andrew G Wilson. Cola: Exploiting com- positional structure for automatic and efficient numerical linear algebra. Advances in Neural Information Processing Systems, 36, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO; Common Objects Context. URL http:://axiii...tt/ea// http://aarxiv.orr//ab//11565 httg//bb/1400..003
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. URL htRp:/tt://arxiv.orr
abs/ 12598
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. URL http://arxiv org/ abs/1801 01401
Hyungjin Chung: Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving Diffusion Models for Inverse Problems using Manifold Constraints, October 2022. URL httttt:////axii..rrrttt// /bs
2206. 00941 arXiv:2206.0094) [cs, stat].
Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Mare L. Klasky, and Jong Chul Ye. Diffusion Posterior Sampling for General Noisy Inverse Problems, February 2023. URL htttp:////ar org/ bogg/abs/200 2209. 14687 arXiv:2209.1466 [cs, stat].
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
Marc Finzi, Anudhyan Boral, Andrew Gordon Wilson, Fei Sha, and Leonardo Zepeda-Núñee User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems, June 2023. Utthttp://arxiv..coo wtwpa/wwwww..rrga/./...oo. 07526 arXiv:2306.07526
[es].
Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion, December 2023. URLphttp:/arrii htgg/abs/2312.. 00852 trXiv:2312.00885 [cs, stat].
12

---

A Gradient Ascent (GA) does not sample
It is possible to reparameterize the Gradient Ascent (GA) procedure from Poole et al. [2022] and Wang et al. [2022) in a way that resembles solving the PF-ODE by carefully selecting optimizer hyperparameters and stopping times. However, this reparameterization is highly specifie. The crucial difference lies in the interpretation of the trajectory: in GA, the path taken is typically irrelevant as long as it converges to the critical point, allowing us the flexibility to multiply the gradient by any positive semidefinite (PSD) matrix without affecting the solution.
For the PF-ODE, however, the trajectory is everything. The stopping time, weighting terms, and score evaluations must be precisely aligned ensure that the solution looks like typical sample. Even small deviations in the path, like evolving the PF-ODE for too long or too short, can lead to either oversmoothed samples (with too much entropy removed) or under-resolved. noisy samples. Thus, while GA might superficially look like a discretized version of the PF-ODE, the underlying processes are inherently distinct: one finds the mode, and the other describes the full trajectory required to generate samples from the distribution. In the case of SDS and SJC, they both use Adam as the optimizer, ignoring any of the nuanced considerations needed to transform the PF-ODE into the corresponding optimization procedure.
In practice, this distinction means that while both SDS and SJC may yield plausible diffreps at high classifier-free guidance levels, they tend to falter when producing coherent samples under lower CFG weights-pparticularly for complex, multimodal distributions with many degrees of freedom. High CFG levels may mask these limitations by strongly biasing the output towards the mode, which aligns more closely with the GA approach, but at the cost of reduced sample diversity and potentially missed details captured by the full distribution.
The problem with mode-seeking To appreciate the limitations of using the mode as proxy for sampling, we consider the multi-modal and the high-dimensional settings.
In the multimodal setting, the sample distribution may contain several distinct high-density regions. However, mode-secking algorithms focus on only one of these regions, sacrificing sample diversity.
In low-dimesiionl spaces, the mode often resembles a typical sample. However, this intuition breaks down in high-dimensional spaces (d >> 1), such as the space of images. This counterintuitive behavior can be explained using the thin-shell phenomenon. For high-dimensional standard Gaussian distribution, samples are not concentrated near the mode (the origin), as one might expect. Instead, they predominantly reside an exponentially thin shell at the boundary of v Vd-radius ball centered on the mode. This phenomenon explains why the mode can be an anomalous and atypical point in high-dimensional distributions.
As an illustrative example, consider sampling a normalized "pure-noise" image. Despite having a mode of O, would almost never expect sample uniformly gray image. This provides some insight as to why the mode of distribution with several degrees of freedom will lack the quality and details present only samples from the thin shell.
B General conditioning
While RePaint can be used to condition the reverse trajectory, tends to be less generalizable for several tasks. For more general conditioning including inverse problems Baademss-MGG [Chung et al. 2022] and its extension DPS [Chung et al., 2023 have gained popularity. These methods add a corrective term to the score, encouraging reverse steps to minimize constraint violations for the predicted final sample in according to the Tweedie formula. MCG and DPS have seen considerable
application and extension to broader settings [Bansal et al.] 2023). Finzi et al. (2023) showed this
correction to be an asymptotically exact approximation of V log? (constraint) zz), while Rout et al. 2003 derive the relevant term from Bayes' rule and higher-order corrections.
C PF-ODE Pullback Derivation and Geometric Details
This section provides the details for the derivation of Eg. 2 using differential geometry.
13

---

First, we provide a full expansion of Eq. 1 For any point in our sample space E X, we can define a Riemannian metric M, that describes the local geometry of the space around z. Given tangent vectors y, at point z, we define the inner product using this metric (y, 2) M, = y y M, where M2 represented using positive-defintte matrix. We implicitly use the Fuclidean metric for the image space X, which corresponds to the identity matrix M2 Ia. Væ X. However, in curved spaces such as the parameter space θ the corresponding Riemannian metric is not necessarily the identity and can vary based where it is evaluated.
Here, we use matrix notation rather than Einstein summation or coordinate- notation for familiarity within the machine learning context. However, we note that matrix notation can obscure some important aspects of differential geometry. For example, while the components of the Euclidean metric in the Euclidean coordinate basis form the identity matrix, the metric is not the identity function, as transforms between two distinct vector spaces. As such, when using the matrix notation, it becomes very important to keep track of the types of objects rather than just their values.
Because we use the Euclidean metric for the image space, its role in [q. is hidden. Explicitly incorporating the metric into the process, we get the following expanded metric PF-ODE
dæ dt (())(())M
V log dP, dPi//da::
(6)
where P. is the probability measure at time and A is the Lebesgue measure on X, and dPP. dd2 is the Radon-Nikodym derivative. Notice that when we use M2 = Ta, Vx E X, this equation equivalent
Eq. I If we pull back the dynamics of this process through the render function f: X + θ to the parameter space θ, we must pull back both the score function and the metric M, to get the pulled back the vector field fo de pullback f":
The different object types transform as follows with the
dt
Scalar Functions
xEC()) (0*))(0)= = s((())
(7)
Cotangent Vectors
WET'X: (f"w)(0) J w((())
(8)
Metric Tensor
M METXOTTX (F)))()) J MS(o
(9)
Tangent vectors
V T (f*b)(0) = (J (JO Moooy(f(O)) (10)
where J the Jacobian of f evaluated at 0. For a scalar function sar), the gradient Vs(x) is co-vector field (also known as differential 1 1-frrm) and can be converted into vector field using the inverse metric: (s()). We can see that Eq. 10 can be expressed concisely in terms of the pullback metric f MM) which geometrically corresponds to converting the vector field to a covector field with M, pulling back the covector field (using the chain rule), and then converting back to vector field with the inverse of the pulled back metric. This sequence is illustrated in Fig.
From Eq. 9 the pullback of the Euclidean metric is JJIJ JJJ. J. From these expressions, we can that the form of f da should be:
dr (t)a(t)( M) ( M M M (log /Pp//)x))
dt dx ((t) (oo(t)(J)(M(oJ))--a - (1og doogd.. dP,//d.))
dt
S dx o) logge (on Euclidean X' where M₂ = I).
di
Thus the pullback of the reverse time PF-ODE is
dl
di fr dr di
-
dx di
-
(())(()))))))))))) Tv log P. (f()))
(II)
Change of variables contribution. Some readers may be surprised to see that the Jacobian log determinant does not show up in this transformation. Though somewhat technical, this can be seen by unpacking the Radon-Nikodym derivative d.P:/d,, In general, on a manifold with metric M, the Lebesgue measure is given by d vdee M,ddr, Therefore, when written in terms of the density /p/ (z)),, one has dPP//de Pc(x)/ det M,. When evaluated only in the Euclidean space
14

---

where the coordinates are chosen such that M, = I, this detail can be safely ignored. However, when pulling back the parameter space, these hidden terms matter. The Radon-Nikon MadonnNikodym derivative dPt/dAz is scalar field and is not transformed under the pullback, however pt is a scalar density and satisfies
(f) = pp (f(()) det(
the familiar change of variables formula. When the imput and output dimensions are the same (0( (0) det J. However, det M, is also scalar density and similarly transforms:
f" det M det M det JT M.J det JT
Assembling these components together, that the determinant contribution from the change of variables formula cancels with the contribution from the change of the Lebesgue measure in the sampling ODE.
(Vooo(dP//ddaa) V log ( det
/ /Vloo(dP./dAaa) - J v (log( ( p.) log I V Ma)
f ('Vlo(dPP//OD)) J v (og Pt + log det JJ log det J MzJ)
2
f'V log (og(dd///>a.. = J log
This is why the change of variables term log det JTJ term does not appear in the pullback of the PF-ODE.
D Details on constrained sampling
For our experiments, used the stochastic non-Markovian reverse process from DDIM Song et al. [2022]. For the RePaint steps, we also need to take forward steps to encourage the samples that are "renderable". In this section, we prove the correct form for the forward process looks like
al)) (t) stt) dt) + o(t - dt) (Vor(t- dt) - - dt) + r(t)e,(t)
(12)
Where of) is the standard deviation and can be calculated using att) (1/o²(1)+1 and s(t) is the scale of signal. It can be calculated using s(t) Vaz(1)+I where a(t) is the noise-to-signal ratio. In the main body the text, we let ott) notate the noise-to-signal ratio, and we assume s(f) but that is not necessarily true in the general case.
1
Theorem D.1. Eq. 12 is the correct forward update for the non-Markovian process.
Proof. We will use the notation x(]00) xt) = x0t))= x( - dt, 0) x(t)lx(t (t))))/(tdd))rrrpt)) x(0). and ætt - dt|t, 0) x(t (t))) The same holds if we replace with Eo-
Using the reparametrization trick we can write z=t) x (t s(t) (ttco)tt))) so
In p(=((tt)))) (t(0) +
Using DDIM and the reparameterization trick, we can also write
1
s(t - dt)
x00) Vo dt) =2(
p((((t dt
T(t) z(t dt)
co(t) tc
2
T(t)
T(t)
1
a(t - dt)
V o dt) - (t)
6ol! dl)
c0()) +c
2
7(1)
r(1)
15

---

Now we can use Bayes' theorem
p(x(t|t dt, 0)) = p(a(t - dtlt, 0))p(z(((())
p(a(t dt(o))
2
1
a(t dt)
V dI) (²)
Inp(s(t((t dt,
co(t - ddt)
ca(t)
+ +
-g(t dt)
+
2
T(t)
T(t)
2
1
at - dt)
ct( -Vo dt)
(²(
EDlt dt)
tc
2
T(1)
T(1)
2
o 0²(1 - di)
(a(1) Vo2(t dt) +2(t)
"0(l dt)
+c
2
(t)
att - dt)
Using reparametcrization, we can say that (conditioned on z(0)), z(t)t - dt, 0) s(t)z(0) (t) dt, 0), where
di) - <o(l dt), 02 dt)
72(1)
ca(1|t dt,0) ~N
alt dt)
E Summary of key ideas and algorithm
Our method introduces several crucial innovations for sampling differentiable representations using pretrained diffusion models:
1. True sampling vs. mode-seeking: Unlike SJC and SDS, which primarily find modes of the distribution, our method aims to generate true samples. This distinction is critical for capturing the full diversity and characteristics of the underlying distribution, especially in high-dimensional spaces.
2. Correct pullback of the PF-ODE: We derive the mathematically correct form of the pulled-back Probability Flow ODE, which includes a crucial (J JJ- term. While this term acts as preconditioner optimization- approaches (affecting only convergence rates), it is essential for unbiased sampling in our PF-ODE framework. Omitting this term leads to incorrect results.
3. Efficient implementation: We introduce techniques for efficient implementation, including separated noise handling and a faster suboptimization procedure. These innovations allow for the practical application of method to sample complex differentiable representations.
4. Generalization to coupled images and stochastic functions: Our method extends naturally scenarios involving coupled images and stochastic functions, making applicable to wide range of problems, including 3D rendering and panorama generation.
5. Handling multimodality and constraints: Our sampling approach naturally handles mul- timodal distributions, unlike mode-seeking methods. However, this introduces challenges when the samples are incompatible with the constraints imposed by the differentiable rep- resentation. We address this by conditioning the sampling process with the consistency constraint using an adapted RePaint method.
These key ideas collectively enable our method to generate high-qualigy diverse samples of differen- tiable representations by encouraging the diffusion model maintain consistency with the constraints imposed by the representation.
The pseudocode for the DDRep algorithm using DDIM sampling with RePaint is presented in Alg.
F Additional experimental details
This section presents additional details and runtime metrics for our algorithm with the experimental settings given in section 4
16

---

Algorithm DDRep with RePaint
1: Input: cfg. reverse_
hump_intevvallppmmm jump_len, hamp__epeat.
2: Init: 0 arg ming - (0, . <(T)) ~ (0, /), VT El
3: schedule =
htw.rooeeddiiteoerrreeeesssseeee htrprro__sstppssu_p_iiteerreee jump_len jump_repeat)
4: for (reverse, t) in schedule do
5:
Olangevin 7/ =
(1--00)++
(0)+1
6: Clangevin (0, T langevin
7: Sample Views: π ~ II
8: reverse then
9:
xt() f(0, T7) + (T)))(M)
10:
(T( = É(x((), t, viww__pommp(m)) cfg)
Feett x(t) (T)) V
o iaggoiin ot Af) o(t - flangevin
12: else
Tnext [(0 + o00 langevin < (T)) + cr elnggvin
14:
end if
15:
Update Noise Parameters: e(TT) / T
iangovin <(TT) + Ctangevin
16:
Update Diffrep Parameters: 0 = arg ming [0] T) Tnext + [
17:
end for
cfg scale
0
10
30
100
PSNR f
29.712 + 4.231
29.931 5.767
27.593 + 6.574
23.453
6.397
13.586
4.024
SSIM
0.899 = 0.086
0.896 0.107
0.888 + 0.115
0.826 = 0.145
0.523 = 0.183
LPIPS
0.044 + 0.050
0.053 0.074
0.061 + 0.074
0.098
0.098
0.336 + 138
Table 1:
Image similarity scores (mean + std) for the images produced using method and those produced using the SD reference.
Image SIRENS Sampling batch of eight reference images using SDv1.5 takes 39 seconds. Generating SIRENs using SJC depends on the number of iterations used; running SJC for 3000 iterations generate eight SIRENS takes 694 seconds, corresponding to a per-iteration time of 2.31 seconds. In comparison, sampling a batch of eight SIRENs using our method takes 82 seconds. These timings demonstrate that our method offers significant speed advantage over SJC while producing comparable results to the reference SDv1.5 1.5 samples.
The PSNR, SSIM, and LPIPS scores of our method against the SDv1.5 reference images presented in Table
SIREN Panoramas Generating SIREN panorama with method takes 218 seconds. Fig. 7 shows how using RePaint to condition the sampling procedure can generate more coherent panoramas for low CFG scales. Additional examples of SIREN panoramas generated using method provided in Fig.
3D NeRFs Our algorithm processed a batch of eight views per iteration, which was the maximum capacity for the A6000 VRAM. The generation of each NeRF required approximately 7.2 seconds per NFE (Number of Function Evaluations). We implemented 100 inference steps using the DDIM procedure, with one RePaint forward step for each reverse step. This configuration resulted in 199 NFEs per NeRF, translating to about 24 minutes of sampling time for each NeRF. It is worth noting that run times can fluctuate based on the specific sampling procedure and GPU architecture used. For comparison, generating comparable NeRF using SJC takes approximately 34 minutes for 10,000 steps, based on the code provided by the authors.
17

---

CFG Scale 3.0
No RePaint RePaint
Figure 7: Comparison of landscape panoramas sampled using method. The top panorama is sampled using the RePaint method, while the bottom sampled without RePaint. Both approaches use 460 function evaluations (NFEs) to ensure fairness and a CFG scale of 3.0. The prompt for these panoramas was "Landscape picture of a mountain range in the background with an empty plain the foreground 50mm f/1.8".
Figure 8: Additional SIREN Panoramas with prompts (top) "Urban skyline twilight, city lights twinkling in the distance." (middle) "A futuristic kitchen" (bottom) "Desert oasis, palm trees sur- rounding a crystal-clear pool."
G Future work and limitations
Our work has demonstrated the zero-shot generation of implicit images, panoramas, and NeRFs. The versatility of our approach opens wide range of potential applications, including vector graphics generation, embedding diverse content in scenes through geometric transformations, differentiable heightmaps, and applications using differentiable physics renderers.
While focused NeRFs in this paper, advanced differentiable representations like Gaussian splats and full-scene representations have emerged. Our method should be applicable to these cases, but further work is needed adapt approach to these newer representations.
Looking ahead, one could envision generating entire maps or game environments using diffusion models. However, this would likely require innovations in amortization and strategies for splitting and combining subproblems. In these cases, the function f may be even more restrictive than for SIRENs and NeRFs, underscoring the importance of conditionally sampling the implicit constraint.
Limitations Our method faces two significant limitations. First, the additional steps required for RePaint introduce computational overhead. A priori, we cannot determine how many steps are needed to harmonize the constraint into the diffrep. Transitioning to conditional sampling method like MCG could potentially allow us to skip these extra forward steps and directly integrate the constraint into the solver. However, this requires further investigation to ensure compatibility with method.
18

---

Second, the substantial stochasticity in the Monte Carlo estimate of the pullback score over several views, particularly when the Jacobian of the render function is ill-conditioned, poses a challenge. While increasing the number of sample views could reduce variance in our estimate, it would also slow down our algorithm and consume more memory. Further research into view sampling techniques, such as importance sampling, could help decrease variance without compromising computational efficiency.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We justify all the claims we make in the abstract and introduction throughout the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have a Limitations and conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

All theoretical results explicitly state the full set of assumptions and have complete and correct proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the details needed to reproduce the experiments are provided in the paper. We will also release the code reproduce the experiments.
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer:
[No]
Justification: The code is not currently in state ready for distribution. We will release the code after have some time clean it up.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation for all our metrics and show samples using multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report that all experiments were conducted using single A6000 GPU and we provide the wall clock time needed run each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research in the paper conforms, in every respect, wit the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Our contribution is methodological and we don't ton't know how it may be used.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Wo do not release any data or models in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in the paper properly credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "AoEVuSQEbd", "reviewer": "Reviewer_H3if", "summary": "The present paper introduces a training-free method to sample differentiable representations using pre-trained diffusion models. This is achieved by pulling back the dynamics of the reverse-time process from the image to the parameter space. Moreover, training-free methods for conditional sampling are employed to (approximately) satisfy implicit constraints. Numerical experiments show the performance of the method for images, panoramas, and 3D NeRFs. In particular, it is shown that the method can improve upon the baselines in terms of quality and diversity of generated representations.", "questions": "Why is the score not the gradient of a scalar function?\n\nTypos:\n1. The visualizations in Fig. 1 (right) seem not to match the explanation \"The example images in Figure 1 (right) show that the samples produced by our method are almost indistinguishable from the reference.\"\n2. The order (top/bottom) in the caption of Fig. 4 is wrong.", "limitations": "Only a few limitations are mentioned and it would be good to enumerate potential failure cases of the proposed method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed framework is versatile and can be used for various differentiable representations. The presented numerical experiments show that the resulting methods can improve quality and diversity compared to other training-free baselines.", "weaknesses": "1. Baselines and related works: Further discussion of related works and additional numerical comparisons are needed.\n    - It would be helpful to provide further discussion on related works, such as Zero-1-to-3, HiFA, Magic123, LatentNeRF, Fantasia3D, ...\n    - While the presented results are promising, additional experiments and visualizations would strengthen the validation of the method.\n    - A more detailed comparison to methods requiring fine-tuning/training could provide a clearer picture of the method's performance. In particular, it would be good to add comparisons in terms of (overall) runtime/NFEs.\n    - Why are there no PSNR/SSIM/LPIPS numbers in Table 1 for the baselines, e.g., SJC?\n    - While DreamFusion is similar to SJC, it would still be interesting to add empirical comparisons.\n\n2. Presentation and theory: Further details and explanations could be provided (e.g., also in the appendix).\n    - The geometric perspective and pullback operation might be challenging for readers not well-versed in these areas. One could improve accessibility by providing more intuitive explanations.\n    - Precise details on how RePaint is adapted to this setting seem to be missing. In this context it would also be good to clarify the connections to other methods for posterior sampling (DPS/ReSample/...).\n    - It would be helpful to add some explanation on the \"separately managed noise object\" $\\epsilon(t)$ or $\\epsilon(\\pi)$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and review. We will address your concerns in turn. \n\nWe have added a discussion of additional 3D asset generation from image diffusion methods like Zero-1-to-3, HiFA, Magic123, LatentNeRF, and Fantasia3D, which you mentioned. These works build off the Dreamfusion (SDS) / SJC method by adding additional inputs, finetuning, or regularization to improve the generation quality or by expanding the inputs for generating 3D assets. Zero-1-to-3 builds off of SJC using the same form of the score chaining, but with a score function fine-tuned to make use of a single input real image and additional view information. Magic 123 uses Zero-1-to-3 with some additional priors based on the 3D information. Fantasia3D separates geometry modeling and appearance into separate components, using SDS to update both. HiFA adds a different schedule and applies regularization to improve SDS. Finally, LatentNERF also uses SDS but parametrizes the object directly in the latent space of the stable diffusion autoencoder, rendered not to RGB values but instead into the dimensions of the latent space. These methods use the SDS/SJC backbone method for sampling/mode finding. In contrast, our work is focused on providing a more faithful sampling procedure to replace SDS/SJC for 3D generation and broader differentiable function sampling.\n\nSimilarity metrics for SJC: Initially, we did not report PSNR, SSIM, and LPIPS for SJC because the generations have little relation to the ones generated by the vanilla diffusion model with the same seed. We have added these in the table below. Notice that they are far, far worse than our method (a better comparison is a measure of distributional similarity like KID, which we report in Figure 1). This is because SJC does not follow the same trajectory as the PF-ODE, so even if it were performing sampling, the samples would not look identical to those produced in the image space. The PSNR, SSIM, and LPIPS scores are used to compare images with identical content but with different compression levels; they are not designed to compare distributions of images.\n\nTable for metrics on SJC\n|  CFG  |          0 |         3 |        10 |        30 |       100 |\n|:------|-----------:|----------:|----------:|----------:|----------:|\n| psnr  | 8.50652    | 8.04771   | 7.13961   | 6.14309   | 5.54287   |\n| ssim  | 0.00938183 | 0.0419226 | 0.0988217 | 0.0973837 | 0.0843486 |\n| lpips | 1.10491    | 0.982072  | 0.855826  | 0.813387  | 0.787548  |\n\nBenchmarking Dreamfusion: SJC and Dreamfusion (SDS) are extremely similar in methodology when accounting for the conversion between the denoising model and the score function. The key differences are in the weighting function that the expectation is taken over, the optimization and sampled times in the schedule, and regularizers added for the 3D case. The SJC paper includes a section discussing the differences between the methods and provides a qualitative assessment of the assets generated by both methods.\n\nPrecise details of how RePaint is applied: The most straightforward approach to encourage the pullback of the PF-ODE to produce intermediate samples that are representable by $f$ is to adapt the RePaint method (designed for inpainting) to this consistency constraint. RePaint takes advantage of the complete Langevin SDE diffusion process instead of the PF-ODE we have been considering here. It works by intermixing several forward and reverse steps in the schedule. Since RePaint requires a stochastic process to apply the conditioning, we use the DDIM \\citep{song_denoising_2022} sampling procedure with $\\eta=0.75$ for both the forward and backward steps.\n\n$\\epsilon(t)$ and $\\epsilon(\\pi)$: Following the pulled-back PF-ODE we can find $\\theta_t$ so that $f(\\theta_t)$ represents a sample $x(t)\\sim p_{0t}(x(t)|x(0)) = \\mathcal N(x(t);x(0),\\sigma^2(t)I)$. One limitation of this approach is that $x(t)$ is noisy, and differentiable image or 3D representations have a hard time expressing noise. In these settings, $J$ is ill-conditioned, leading to poor performance and long convergence times.\n\nTo address this issue, we can factor $x(t)$ into the noiseless signal and noise using the reparameterization of the perturbation kernel $x(t) = \\hat x_0(t) + \\sigma(t)\\epsilon(t)$. If we let $\\epsilon(t) = \\epsilon$ be constant throughout the sampling trajectory and start with $\\hat x_0(T) = 0$, we can update $\\hat x_0$ using $\\frac{d\\hat x_0}{dt} = \\frac{dx}{dt} - \\dot\\sigma(t)\\epsilon$. Using this decomposition, we let $f(\\theta_t)$ represent the noiseless $\\hat x_0(t)$ instead of $x(t)$, which makes it much more likely that $J$ is much better conditioned.\n\nQuestion: Why is the score not the gradient of a scalar function?\nHere, we use scalar in the differential geometry sense of being a scalar quantity on the given manifold, not an object that has a value dependent on the chosen coordinate chart. For continuous probability distributions, the probability density function varies not just on the chosen point but also on the given coordinate chart, as made evident by its changing value under a change of variables (where the additional $\\det J$ term arises). Informally, this can be seen from the fact that $\\int_R p_X(x)dx = \\int_R p_Y(y)dy$, but because of the changing volume element $p_X\\ne p_Y$. In contrast, a scalar function will have the same value when evaluated at the same point but in two distinct coordinate charts.\n\nWe also agree that the method section could also be explained better. We have provided a revised method section in our response to Reviewer wR2k. We hope this more intuitive description of our method improves your consideration of our paper.\n\nWe thank you for your time spent reviewing. We have invested substantial effort in addressing your concerns and improving the quality of the paper, and we ask that you consider adjusting your score in light of our response. Please let us know if you have additional questions we can help answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iLOF1MpQg4", "reviewer": "Reviewer_BA4f", "summary": "This paper introduces a novel, training-free method to sample through differentiable functions using pretrained diffusion models.", "questions": "I do not have questions at this stage.", "limitations": "The authors have discusses some limitations.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It sounds a general method that could apply to many different scenarios.", "weaknesses": "More systematic evaluation of the method in addition to image examples would be great.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. Please see the common response for additional SIREN panoramas. Let us know if there are any remaining concerns we might be able to address.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3fSd8YNBi", "reviewer": "Reviewer_wR2k", "summary": "The paper introduces a method for sampling a differential representation using a pre-trained diffusion model. Instead of sampling in the image space, the authors propose sampling in the parameter space of differential representations by 'pulling back' the probability distribution from the image space to the parameter space of the differential representation.", "questions": "- The two main relevant papers (Poole et al. [2022] and Wang et al. [2022]) are not discussed in appropriate detail:\n  - The authors claim equivalence between the methods without providing a detailed argument or a citation where the argument is made.\n  - Lines 93-94 suggest that these methods act as \"mode-finding algorithms,\" however, a discussion is lacking.\n\n- The arguments in Section 3.1 (which I believe is the main argument of the paper) are hard to follow for a non-expert:\n  - What is the relevance of Equations 3-5?\n  - How are Equations (6) and (7) derived?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a novel method for sampling a differentiable representation for a pre-trained diffusion model, which is highly relevant.\n- Their method appears to be well-principled.\n- Although their results are limited, they seem promising.", "weaknesses": "- The technical part of the paper is difficult to follow.\n- Comparison to the state-of-the-art is provided only in Section 5.1, where a differential representation is not required.\n- The code is not shared at this stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and review. We will address your concerns in turn. \n\nSJC and Dreamfusion (SDS) are extremely similar in methodology when accounting for the conversion between the denoising model and the score function. The key differences are in the weighting function that the expectation is taken over, the optimization and sampled times in the schedule, and regularizers added for the 3D case. We show how the SJC and SDS objectives are identical up to a weighting term and delineate the major differences between the methods and how they compare to our method in the common response.\n\nReliable metrics for generative models are primarily image-based. This is why we used the SIREN image diffrep to evaluate our method quantitatively. While it is possible to generate NeRFs and SIREN panoramas using the other methods, comparing them would be infeasible since there are no good techniques to compare distributions of general diffreps yet.\n\nWe will make the code public in the final version of the paper.\n\nWe have significantly changed the technical description of our methodology to make it simpler and more intuitive. We hope you find this new description clearer.\n\nThe score model associated with the noise predictor implicitly defines a distribution on the data space $\\mathcal X$, and the reverse time PF-ODE provides a means to sample from that distribution. \nIn this section, we will show how to use the score model and the PF-ODE to sample the parameters of a differentiable representation (diffrep) so that the rendered views look like samples from the implicit distribution.\n\nIn SJC and DreamFusion, they derive their parameter updates using the pullback of the sample score function through the render map $f$. Explicitly, this pullback is obtained by using the chain rule $\\tfrac{d(\\log p)}{d\\theta} = \\tfrac{d(\\log p)}{dx}\\tfrac{dx}{d\\theta}$. The pullback score is then used __as is__ to perform gradient ascent in the parameter space. However, since $p$ is a distribution and not a simple scalar field, the change of variables formula requires an additional $\\log \\det J$ for the appropriate score function in the parameter space, where $J$ is the Jacobian of $f$. Carefully examining this approach through the lens of differential geometry reveals even deeper issues. \n\nFrom differential geometry, recall that it is a vector field (a section of the tangent bundle $T\\mathcal X$), not a differential form (a section of the cotangent bundle $T^*\\mathcal X$), which defines the integral curves (flows) on a manifold. To derive the probability flow ODE in the parameter space, we must pull back the vector field $\\frac{dx}{dt}\\in T\\mathcal X$ to the parameter vector field $\\frac{d\\theta}{dt} \\in T\\Theta$ using the render map $f$. The pullback of the vector field through  $f$ is given by \n$(f^* \\frac{dx}{dt})|_\\theta = (J^\\top J)^{-1}J^\\top \\frac{dx}{dt}|\\_{f(\\theta)},$ and thus the pullback of the probability flow ODE is\n\n\\begin{equation}\n    \\frac{d\\theta}{dt} = -\\dot\\sigma(t)\\sigma(t)(J^\\top J)^{-1}J^\\top\\nabla \\log p_t(f(\\theta)).\n\\end{equation}\n\nWe note that this is in contrast to the SJC and DreamFusion score. The confusion arises when comparing the types of elements that are pulled back. While the left-hand side of the PF-ODE $\\frac{dx}{dt}$ is a vector field, the score function $\\nabla \\log p_t(x(t))$ on the right-hand side is a covector field, i.e., it is a differential form. Pulling back the score function as a differential form correctly yields $J^\\top\\nabla \\log p_t(f(\\theta))$, the term used in SJC. The problem is that hidden in PF-ODE is the use of the (inverse) Euclidean metric to convert the differential-form score function into a vector field to update the parameters. In canonical coordinates, the Euclidean metric is the identity. As a result, the components of the score function do not change when they are transformed into a vector field by the Euclidean metric. Therefore, we can safely ignore the metric term in the original PF-ODE formulation for $\\mathcal X$.\n\nIf we want to convert the pulled-back score function into the corresponding pulled-back vector field, we need to use the pulled-back Euclidean metric inverse given by $(J^\\top J)^{-1}$. When we use this metric, we get the same pulled-back form of the PF-ODE as given above.\n\nA more informal way of seeing why the correct update must look like the pulled-back ODE we have derived here rather than that given by SJC is to look at the case where the number of input and output dimensions is the same. From the chain rule, we have $\\frac{d\\theta}{dt} = \\frac{d\\theta}{dx} \\frac{dx}{dt}$.\nWe can compute $\\frac{d\\theta}{dx}$ using the inverse function theorem $\\frac{d\\theta}{dx} = \\left(\\frac{dx}{d\\theta}\\right)^{-1} = J^{-1}$, where $J$ is the Jacobian of $f$. So we arrive at $\\frac{d\\theta}{dt}=J^{-1}\\frac{dx}{dt}$. When $f$ is invertible, this equation is equivalent to the one given above. For non-invertible $f$, we can interpret the pullback as the solution to the least squares minimization problem $\\min\\left\\|J\\frac{d\\theta}{dt} -\\frac{dx}{dt}\\right\\|^2$ for $\\frac{d\\theta}{dt}$.\n\nWe thank you for your time spent reviewing. We have invested substantial effort in addressing your concerns and improving the quality of the paper, and we ask that you consider adjusting your score in light of our response. Please let us know if you have additional questions we can help answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4dmBjU5cKE", "reviewer": "Reviewer_za3b", "summary": "This paper introduces a zero-shot method to sample neural representations with pre-trained diffusion models. By pulling back the measure over the data space through the representation, the authors express the PF ODE in the parameter space. Solving this ODE can directly provide parameter samples of the representation. The authors also discuss cases where the representation needs to model coupled images. In this scenario, the PF ODE is expressed by expectation and can be solved following the same principle. This method offers random samples rather than returning the mode, which can yield results with higher diversity compared to baselines without sacrificing performance. The method is also compatible with RePaint, which can ensure the implicit constraint (e.g., the inductive bias of the representation or practical consistency constraint).", "questions": "I have two more questions regarding the experiments. \n1. In Fig 1 (right), you compare your method with SJC and claim that \"the samples produced by our method are almost indistinguishable from the reference.\" However, I do not necessarily agree with this claim. Your method yields much smoother images than SD. Even though I mainly suspect this may be due to SIREN's inductive bias, it is visible and distinguishable. Additionally, it seems that your method provides very different outcomes with CFG 30, while results by CFG 3/10/100 are pretty similar. Do you have any explanation for this? Is it a sign that this approach is not robust?\n2. The PF ODEs in all experiments need to meet some implicit constraints due to the inherent bias of INRs. However, you only apply RePaint for Panorama SIRENs to ensure the consistency of the generated panorama. Do I understand this correctly? Is there a difference if you ensure the implicit constraints for other experiments? Is it possible to compare the results with and without ensuring the constraint?\n\n\nIn general, I appreciate the contribution of this work and would be happy to raise my score if my questions are adequately addressed.", "limitations": "I did not find discussions on limitations even though Sec 6 is called Limitations and Conclusion. I am unsure if this work has no limitations, such as runtime. Could the author please clarify this?", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This work introduces a zero-shot method to sample neural representations by pre-trained diffusion models. The method is fully zero-shot and does not rely on the specific form of the representation. Therefore, it may gain great potential in downstream tasks where training is difficult, or the representation is highly restrictive.\n2. This method can generate samples with significantly higher diversity compared to the baselines, and can ensure consistency constraint using RePaint, improving global coherence and quality of generation.\n3. The idea to handle the Jacobian matrix by approximating each ODE step using an optimization task is cute and neat.", "weaknesses": "My primary concern is its runtime. If I understand correctly, you need to run the optimization for several iterations for each data point and each Euler step. And for tasks like NeRF, where the PF ODE involves expectations, this optimization also involves an MC estimator. However, this paper only reported the runtime for 3D NeRF experiments. I am curious on \n\na. what is the runtime for other experiments? What is the runtime for the baselines? Is this method significantly more expensive than others?\n\nb. How many samples do you use to estimate the expectation? Does it contribute to the runtime? Will reducing the number of samples increase the variance and lead to suboptimal performance?\n\nI would still appreciate this approach even though it may be more expensive. However, reporting the runtime and comparing it with the previous baselines can help us understand this method better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and review. We will address your concerns in turn. \n\nExperiment details: As mentioned in the paper, generating a NeRF with our method takes around 24 minutes for 199 NFEs. We used as many samples as possible in the 40GB VRAM of an NVIDIA A6000; our experiments were eight samples. Generating a comparable NeRF using SJC takes around 34 minutes for 10000 steps using the code they provide. Generating eight batched SIREN images using our method takes 82 seconds. Generating a SIREN Panorama with our method takes 218 seconds. \n\nAs you correctly pointed out, Figure 1 in the paper had some issues (the images were out of order). We have briefly described the problems in the common response and attached the corrected Figure 1 in the accompanying PDF.\n\nWe use RePaint for all our experiments, including the SIREN image and NeRF experiments. Without the RePaint method, the results are much less compelling. In fact, the NeRF experiments diverge without it. The INR implicitly defines the constraint, so it is impossible to remove the constraints from the INR. However, as a proxy for the constraint, one could consider using the CFG scale, where a high CFG corresponds to a stronger constraint because it enforces the renders from the INR to look a specific way. On the other hand, low CFG means more flexibility in the kinds of INR we can sample. Figure 3 shows the results of ablating RePaint on SIREN Panoramas with a high and a low CFG.\n\nWe thank you for your time spent reviewing. We have invested substantial effort in addressing your concerns and improving the quality of the paper, and we ask that you consider adjusting your score in light of our response. Please let us know if you have additional questions we can help answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Firstly, we would like to thank all the reviewers for their thoughtful comments and feedback on our submission. We appreciate the opportunity to address your concerns and clarify aspects of our work.

We want to reiterate the contribution of our work as a superior method to perform true sampling of differentiable representations (diffreps) using a pretrained diffusion model instead of merely mode finding as performed by DreamFusion and SJC. This allows our method to generate diverse and high-quality diffreps for a given prompt, even at low CFG levels. Also, we want to highlight that our method is training-free compared to methods like ProlificDreamer. This allows it to be applied directly with any score estimator regardless of architecture, making our contribution considerably more versatile and applicable to various domains and modalities.

We want to make an important note that may have given the impression that our method is not performing as well as it actually does in practice. In Figure 2, we have compared the raw image samples generated by following the PF-ODE in image space to the SIREN renders generated with our method and SJC. This figure is a substantively more faithful comparison of the generation abilities of the methods than Figure 1 (right). The results in Figure 1 (right) are presented out of order, giving the impression that there is no clear winner between the two methods and introducing the apparent tension with the statement, "samples produced by our method are almost indistinguishable from the reference." We apologize for this oversight and have corrected the figure, attaching it to the material below. The new Figure 1: (Corrected) demonstrates the clear improvement of our method, particularly at low CFG levels when the distinction between mode and typical samples is most relevant.

We provide a more detailed explanation of why DreamFusion and SJC are practically identical and why they perform mode-finding instead of sampling next.

In DreamFusion, they perform gradient ascent using $\mathbb{E}\_{t,\epsilon}[w(t)J^\top(\hat\epsilon(f(\theta) + \sigma(t)\epsilon)-\epsilon)],$ derived from the denoising objective, where $J$ is the Jacobian of the differentiable render function $f$ and $\hat{\epsilon}$ is the noise predictor. Score Jacobian Chaining (SJC) performs gradient ascent using $\nabla \log p(\theta):= \mathbb{E}_{t,\epsilon}[J^\top \nabla\log p_t(f(\theta)+\sigma(t)\epsilon)].$ If we rewrite the DreamFusion objective $\mathbb{E}\_{t,\epsilon}\big[w(t)J^\top (\hat{\epsilon}(f(\theta)+\sigma(t)\epsilon)-\epsilon)\big]$ using the Tweedie formula, we get $\mathbb{E}\_{t,\epsilon}\left[\tfrac{w(t)}{\sigma(t)}J^\top \nabla \log p_t(f(\theta)+\sigma(t)\epsilon)\right].$

This expression is identical to the $\nabla \log p(\theta)$ term from SJC if we let $w(t) = \sigma(t)$ for all $t$.

Both approaches estimate the objective using Monte Carlo sampling. In addition, SJC also uses a custom sampling schedule for the $t$s, which can be interpreted as gradient annealing to align the implicit $\sigma(t)$ of the diffrep with the $t$ used to evaluate the score function. Following the gradients to convergence leads to a critical point in the $\log p(f(\theta))$ landscape, finding a (local) maximum or mode. Neither procedure has a set stopping time, and running the PF-ODE using the parameter score function suggested by these methods would fail to produce samples from the distribution (consider the procedure on Gaussian data as a concrete example). Furthermore, while both methods can produce passable diffreps at high classifier-free guidance (CFG) levels, they struggle to produce coherent diffreps when the CFG weight is low, especially if the distribution is multimodal or has more degrees of freedom.

To appreciate the limitations of using the mode as a proxy for sampling, we need to consider two scenarios: the multi-modal setting and the high-dimensional setting. In the multi-modal setting, the sample distribution may contain several distinct high-density regions in the multi-modal setting. However, mode-finding algorithms typically focus on only one of these regions, potentially sacrificing sample diversity. This lack of diversity can be seen in the SJC samples (last row) of Figure 2.

Although the mode might look like a typical sample in low dimensions, the mode is anomalous when the sample space is high dimensional $d \gg 1$ (e.g., the space of images). This fact can be intuitively understood as a consequence of the thin-shell phenomenon, which states that while low-dimensional standard Gaussian samples concentrate around their mode, high-dimensional Gaussian samples predominantly reside in an exponentially thin shell around the boundary of a ball centered on the mode with a radius $\sqrt{d}$. As an illustrative example, consider sampling a normalized pure-noise image. Despite having a mode of $0$, we would almost never expect to generate a uniformly gray image. This provides some insight as to why the mode of a high-dimensional distribution will lack the quality and details present only in the samples from the thin shell.

We have also added more panorama images in the supplementary PDF.

Once more, we would like to highlight the uniqueness of our contribution as the true pullback of the diffusion process to the diffrep parameter space. Unlike fine-tuning methods like ProlificDreamer or other training-free methods, our approach provides a means to efficiently apply the dynamics of the diffusion process directly in the parameter space through the pullback to sample high-quality diffreps.

We thank the reviewers for their valuable insights and hope our responses have addressed their concerns. We believe that our work significantly contributes to the distillation of diffusion sampling space, and we hope this response alleviates any concerns the reviewers have.

Meta-review: This paper proposes a zero-shot method for sampling implicit representations from pretrained diffusion models, which could in principle allow for better coverage of the full distribution, whereas prior methods might be more susceptible to mode collapse. The reviewers generally found the paper interesting and the contribution promising, but identified limited scope of experiments (for example the need for comparison to more baselines, and to validate the diversity claims in further settings) as a weakness, as well as some issues with clarity or errors in presentation. After discussion, some of these concerns were addressed — particularly presentational ones — but the concerns about the coverage of the experiments remain. Overall, the conceptual contribution seems potentially valuable and impactful, but it would need a more convincing and comprehensive demonstration of the benefits to achieve optimal impact.
Decision: Accept (poster)