Diffusing Differentiable Representations

Authors: yash  savani
marc  finzi
j  zico kolter
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:35:15

Abstract: We introduce a novel, training-free method for sampling *differentiable representations* (diffreps) using pretrained diffusion models. Rather than merely mode-seeking, our method achieves sampling by "pulling back" the dynamics of the reverse-time process—from the image space to the diffrep parameter space—and updating the parameters according to this pulled-back process. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects. Our method yields diffreps with substantially **improved quality and diversity** for images, panoramas, and 3D NeRFs compared to existing techniques. Our approach is a general-purpose method for sampling diffreps, expanding the scope of problems that diffusion models can tackle.

Main paper: 
 introduction: 
 Diffusion models have emerged as a powerful tool for generative modeling [Ho et al., 2020 , Song et al., 2022 , 2021 , Karras et al., 2022 , Rombach et al., 2022] , and subsequent work has extended these models to generate complex objects-such as 3D assets, tiled images, and more. Although such approaches typically require training or at least fine-tuning of the diffusion models for these new modalities [Wang et al., 2023 , Luo et al., 2023] , two notable exceptions, Wang et al. [2022] and Poole et al. [2022] , have developed methods for training-free production of 3D objects by directly using an image-based diffusion model. Both methods work by optimizing a differentiable representation (diffrep)-in this case, a Neural Radiance Field (NeRF) [Mildenhall et al., 2020] -to produce rendered views consistent with the output of the image-based diffusion model. Unfortunately, the nature of both these methods is that they optimize the diffrep to produce the "most likely" representation consistent with the images; that is, they perform mode-seeking rather than actually sampling from the diffusion model. This results in overly smoothed outputs that lack detail and do not reflect the underlying distribution of the diffusion model.In this paper, we present a novel method for sampling directly in the diffrep space using pretrained diffusion models. The method is training-free, can handle arbitrary diffreps, and performs true sampling according to the underlying diffusion model rather than merely mode-seeking. The key idea of our approach is to rewrite the reverse diffusion process itself in the diffrep parameter space. This is achieved by "pulling back" the dynamics of the reverse time process-from the image space to the parameter space-and solving a (small) optimization problem, implied by the pulled-back dynamics, over the parameters for each diffusion step. To further encourage solver convergence, we identify constraints that the diffrep induces on the samples and use these constraints to guide the reverse process to generate samples from the high-density regions of the diffusion model while satisfying the constraints along the sampling trajectory.Our experiments use a pretrained image-based diffusion model (Stable Diffusion 1.5 [Rombach et al., 2022] ) to generate samples from various classes of diffreps. For example, by sampling SIREN representations [Vincent, 2011] with wrap-around boundary constraints, we can sample total 360degree panorama scenes, and by sampling NeRF representations [Mildenhall et al., 2020] , we can generate 3D models of many objects. In both settings-as well as in baseline comparisons on simple SIREN-based (non-panorama) image generation-our approach substantially outperforms previous methods such as Score Jacobian Chaining (SJC) [Wang et al., 2022] . Though the problem setting is considerably more difficult without fine-tuning or retraining, sampling diffreps using pretrained diffusion models with our method substantially improves and extends the state-of-the-art in training-free generation methods.2 Related work and preliminaries 2.1 Differentiable representations (diffreps) Diffreps are a powerful tool for representing complex scenes or manifolds using parameterized differentiable functions to map coordinates of the scene into a feature space that encodes the properties of the scene at those coordinates. Popular instantiations of diffreps include: SIRENs [Sitzmann et al., 2020] , which implicitly model an image using an MLP with sinusoidal activations to map 2D (x, y) pixel coordinates into RGB colors, and NeRFs [Mildenhall et al., 2020] , which implicitly model a 3D scene using an MLP that transforms 3D (x, y, z) voxel coordinates with view directions (ϕ, ψ) into RGBσ values. We can render images from different views of the scene by numerically integrating the NeRF outputs along the unprojected rays from the camera.Faster or alternative diffreps also exist for 3D scenes-such as the InstantNGP [Müller et al., 2022] model or Gaussian splats [Kerbl et al., 2023] -although in this work we focus on the basic NeRF architecture. Many other kinds of visual assets can also be formulated as diffreps. For example, we can use diffreps to model panoramas, spherical images, texture maps for 3D meshes, compositions of multiple images, scenes from a movie, and even the output of kinematic and fluid simulations.Many interesting diffreps can be used to render not just one image from the scene but multiple coupled images or even a distribution over images. Given a diffrep, parameterized by θ ∈ Θ, for a scene-such as a SIREN panorama or a NeRF-we can render an image of the scene from a view π ∈ Π using a differentiable render function f (θ, π) = image ∈ X . To accommodate the multiview setting in our discussion, we consider the "curried" Haskell form of the render function f : Θ → (Π → X ), where f (θ) : Π → X is a map itself from a view π ∈ Π to f (θ)(π) = f (θ, π) = image ∈ X .In the case of NeRFs or SIREN panoramas, the view π is a continuous random variable drawn from a distribution that, with abuse of notation, we will also call Π. To formalize this, let H ⊆ X Π be a vector space of functions from Π to X . With this definition, we can write the signature of f as f : Θ → H. Although H is usually larger than X (and can be even infinite in some cases), we can simplify our notation by equipping H with an inner product to make it a Hilbert space. This allows us to use familiar matrix notation with H and hide the view dependence of f (θ) in H.To complete this formalization, we lift the inner product on X to define an inner product on H:∀h, g ∈ H : ⟨h, g⟩ := E π∼Π [⟨h(π), g(π)⟩] = E π∼Π [h(π) ⊤ g(π)].This formulation allows us to handle the multiview nature of NeRFs and SIREN panoramas in a mathematically rigorous way while preserving the convenient notation.The partial application of f returns an entire view-dependent function f (θ) ∈ H. Given a set of images(x i ) i∈[N ] of the scene from different views (π i ) i∈[N ] , the standard diffrep task is to find a θ ∈ Θ that minimizes L(θ) = i∈[N ] ∥f (θ)(π i ) -x i ∥.Because the diffreps and f are both differentiable, this can be accomplished using first-order solvers, such as gradient descent or Adam.For pedagogical convenience, we identify H with X in the following sections. This identification simplifies the presentation, provides a more interpretable and intuitive perspective on our method, and is precise when Π is a singleton. Because H is a Hilbert space, the main points of our arguments still hold when we consider a larger Π. We describe the specific changes needed to adapt our method to the general multiview setting when H ̸ = X , in subsection 3.2. 
diffusion models: 
 Diffusion models implicitly define a probability distribution via "reversing" a forward noising process. While many different presentations of image diffusion models exist in the literature (e.g., DDPM [Ho et al., 2020] , DDIM [Song et al., 2022] , Score-Based Generative Modeling through SDEs [Song et al., 2021] , EDM [Karras et al., 2022] ), they are all equivalent for our intended purposes.Given a noise schedule σ(t) for t ∈ [0, T ] and a score function ∇ log p t (x(t)) for the distribution p t over noisy images x(t) at time t, we can sample from p 0 by first initializing x(T ) ∼ N (0, σ 2 (T )I) and then following the reverse time probability flow ODE (PF-ODE) given in Karras et al. [2022] to transform the easy-to-sample x(T ) into a sample from p 0 :EQUATIONThe PF-ODE is constructed so that the perturbation kernel (conditional distribution) is given by p 0t (x(t)|x(0)) = N (x(t); x(0), σ 2 (t)I). Using the reparameterization trick, we can write this as x(t) = x(0) + σ(t)ϵ, where ϵ ∼ N (0, I).We approximate the score function with a (learned) noise predictor ε(x(t), t)≈ x(t)-E[x(0)|x(t)] σ(t)using the Tweedie formula [Robbins, 1956 , Efron, 2011] : et al. [2022] laid the foundation, demonstrating that generating 3D assets from purely 2D image-trained diffusion models was even possible. In DreamFusion (SDS), they perform gradient ascent using∇ log p t (x(t)) = E[x(0)|x(t)]-x(t) σ 2 (t) ≈ -ε(x(t),t) σ(t) .E t,ϵ [w(t)J ⊤ (ε(f (θ) + σ(t)ϵ, t) -ϵ)], derived from the denoising objective, where J is the Jacobian of the differentiable render function f and ε is the learned noise predictor.Independently, Wang et al. [2022] introduced Score Jacobian Chaining (SJC), which performs gradient ascent using ∇ log p(θ):= E t,ϵ [J ⊤ ∇ log p t (f (θ)+σ(t)ϵ)] with a custom sampling schedule for the ts. The custom schedule can be interpreted as gradient annealing to align the implicit σ(t) of the diffrep with the t used to evaluate the score function.Comparison of methods Using the Tweedie formula, we can rewrite the SDS objective asE t,ϵ w(t) σ(t) J ⊤ ∇ log p t (f (θ) + σ(t)ϵ) 1 .This expression is identical to the ∇ log p(θ) term from SJC if we let w(t) = σ(t) for all t. Both methods follow this gradient to convergence, which leads to a critical point-a local maximum or mode-in the log p(θ) landscape. To approximate the objective, both approaches use Monte Carlo sampling.Both approaches optimize the objective using gradient ascent (GA). While GA on the transformed score resembles solving the PF-ODE-particularly in their discretized forms-the two procedures serve fundamentally different purposes. GA cares only about finding a local maximum in the log p(θ) landscape, regardless of the trajectory taken to get there. In contrast, the PF-ODE follows a specific path to ensure that generated samples are typical of the distribution. For a discussion of the limitations of GA and why it does not produce representative samples from the distribution, see Appendix A.Recent developments Several subsequent works have built upon the SDS and SJC methods by incorporating additional inputs, fine-tuning, and regularization to enhance the quality of the generated 3D assets. Zero-1-to-3 [Liu et al.] expands on SJC by fine-tuning the score function to leverage a single real input image and additional view information. Magic 123 [Qian et al.] further builds on this by incorporating additional priors derived from 3D data. Fantasia3D [Chen et al.] separates geometry modeling and appearance into distinct components, using SDS to update both. HiFA [Zhu et al.] introduces a modified schedule and applies regularization to improve the SDS process. Finally, LatentNeRF [Metzer et al.] utilizes SDS but parameterizes the object in the latent space of a stable diffusion autoencoder, rendering into latent dimensions rather than RGB values. While these methods rely on the SDS/SJC framework for mode-seeking, our work takes a wholly different approach, focusing on developing a more faithful sampling procedure to replace SDS/SJC for both 3D generation and broader differentiable function sampling.Unlike SDS and SJC-which are entirely zero-shot and can be performed with a frozen diffusion model-several other methods have been developed to achieve improved quality and diversity of the generated diffreps, albeit at the cost of additional fine-tuning. While some of these methods require additional data [Zhang et al.] , ProlificDreamer (VSD) [Wang et al., 2023] and DiffInstruct [Luo et al., 2023] are two examples in which diffusion models can be fine-tuned or distilled using only synthetically generated data.VSD specifically addresses the problem of generating 3D assets from a 2D diffusion model using particle-based variational inference to follow the Wasserstein gradient flow, solving the KL objective from SDS in the W 2 (Θ) parameter distribution space. This approach produces high-quality, diverse samples even at lower guidance levels.In our work, we restrict ourselves to the training-free setting. This choice more readily enables application to new modalities and is independent of the score function estimator's architecture.Our method requires generating multiple consistent images of a scene from different views. For instance, we generate various images of a 3D scene from different camera locations and orientations to determine NeRF parameters. These images must be consistent to ensure coherent updates to the diffrep parameters. While the diffrep inherently enforces this consistency, we can improve our method's convergence rate by encouraging the reverse process to satisfy consistency conditions using constrained sampling methods.Several approaches enable constrained sampling from an unconditionally trained diffusion model. The naïve approach of projecting onto constraints [Song et al., 2021] leads to samples lacking global coherence. In contrast, Lugmayr et al. [2022] propose RePaint-a simple method for inpainting that intermingles forward steps with reverse steps to harmonize generated noisy samples with constraint information. For references to more general methods used to constrain the distribution by conditioning the score function, see Appendix B. 
pulling back diffusion models to sample differentiable representations: 
 The score model associated with the noise predictor implicitly defines a distribution on the data space X , and the reverse time PF-ODE from Eq. 1 provides a means to sample from that distribution. In this section, we will show how to use the score model and the PF-ODE to sample the parameters of a differentiable representation (diffrep) so that the rendered views resemble samples from the implicit distribution.SDS and SJC derive their parameter updates using the pullback of the sample score function through the render map f . This pullback is obtained using the chain rule: d(log p) dθ = d(log p) dx dx dθ . The pullback score is then used as is for gradient ascent in the parameter space. However, since p is a distribution-not a simple scalar field-the change of variables formula requires an additional log det J volume adjustment term for the correct pulled-back score function in the parameter space, where J is the Jacobian of f . A careful examination of this approach through the lens of differential geometry reveals even deeper issues.In differential geometry, it is a vector field (a section of the tangent bundle T X )-not a differential 1-form (a section of the cotangent bundle T * X )-that defines the integral curves (flows) on a manifold. To derive the probability flow ODE in the parameter space, we must pull back the sample vector field dx dt ∈ T X to the parameter vector field dθ dt ∈ T Θ using f . The pullback of the vector field through f is given by(f * dx dt )| θ = (J ⊤ J) -1 J ⊤ dx dt | f (θ), Thus, as we derive in Appendix C, the pullback of the probability flow ODE is:EQUATIONWe note that this is in contrast to the score chaining from Poole et al. [2022] , Wang et al. [2022] .The confusion stems from comparing the different types of pulled-back elements. While dx dt in Eq. 1 ⃝ pulling back ∇ log p(x) via the chain rule using the Jacobian J, and then 3 ⃝ transforming the pulled back differential form score function into the corresponding vector field using the inverse of pulled back metric (J ⊤ J) -1 . When used in a PF-ODE, SJC and SDS take the bottom path with the chain rule, however they do not complete the path by neglecting the T * Θ → T Θ transformation. (Right) SIREN image renders generated using the PF-ODE schedule with the prompt "An astronaut riding a horse" using the: (a) complete pulled-back dx dt vector field, (b) pulled-back covector field from SJC (omitting step 3 ⃝) J ⊤ ∇ log p(x), (c) Scaled pulled-back covector field from SJC λ = 0.0001.Θ X f T Θ T X T * Θ T * X I 1 J ⊤ 2 (J ⊤ J) -1 3 (J ⊤ J) -1 J ⊤ dθ dt = (J ⊤ J) -1 J ⊤ dx dt dx dt ∇ log p(θ) = J ⊤ ∇ log p(x) ∇ log p(x) (a) (J ⊤ J) -1 J ⊤ dx dt (b) J ⊤ ∇ log p(x) (c) λJ ⊤ ∇ log p(x)is a vector field, the score function ∇ log p t (x(t)) is a covector field-a differential form. Pulling back the score function as a differential form correctly yields J ⊤ ∇ log p t (f (θ))-the term used in SJC. The issue lies with the hidden (inverse) Euclidean metric within Eq. 1, which converts the differential-form score function into the corresponding vector field.In canonical coordinates, the Euclidean metric is the identity. Consequently, the components of the score function remain unchanged when transformed into a vector field. Therefore we can safely ignore the metric term in the PF-ODE formulation of Eq. 1 for the sample space X . However, this is not true for the diffrep parameter space Θ.To convert the pulled-back score function into the corresponding pulled-back vector field, we must use the pulled-back inverse Euclidean metric given by (J ⊤ J) -1 . This yields the pulled-back form of the PF-ODE in Eq. 22 . Fig. 1 illustrates this procedure via a commutative diagram and provides an example of what goes wrong if you use the incorrect pulled-back term, as suggested by SDS and SJC. In the SDS and SJC approaches, the (J ⊤ J) -1 term behaves like a PSD preconditioner for gradient ascent. It may accelerate the convergence rate to the mode but does not fundamentally change the solution. However, this term is critical for the pulled-back PF-ODE since it impacts the entire trajectory and the ultimate sample (see Appendix A for more discussion). For an explanation of why the log det J term is absent in the mathematically correct update, see Appendix C.A more intuitive way to understand why the correct update is given by Eq. 2 rather than SDS and SJC's version is to consider the case where the input and output dimensions are equal. From the chain rule, we have dθ dt = dθ dx dx dt . Using the inverse function theorem, we compute dθ dx = dx dθ-1 = J -1 ,where J is f 's Jacobian. This leads to dθ dt = J -1 dx dt . For invertible f , this equation and Eq. 2 are equivalent. For non-invertible f , we can interpret the pullback as the solution to the least-squares minimization problem arg min dθ dt J dθ dt -dx dt 2 .Separated noise Following the pulled-back PF-ODE in Eq. 2, we can find θ t such that f (θ t ) represents a sample x(t) ∼ p 0t (x(t)|x( 0)) = N (x(t); x(0), σ 2 (t)I). However, this approach has a limitation: x(t) is noisy, and most diffrep architectures are optimized for typical, noise-free images. Consequently, J is likely ill-conditioned, leading to longer convergence times.f f f f f f forward backward Θ σ(t)εFigure 2: The parameters of the diffrep θ t ∈ Θ (torus) are used to render the noiseless signal f (θ t ) = x 0 (t), which are then combined with the noise σ(t)ϵ to generate the noisy sample x(t). We can pull back each step of the reverse diffusion process to update the parameters θ t + ∆θ t .To address this issue, we factor x(t) into a noiseless signal and noise using the reparameterization of the perturbation kernel:x(t) = x 0 (t) + σ(t)ϵ(t). By letting ϵ(t) = ϵ remain constant throughout the sampling trajectory and starting with x 0 (T ) = 0, we can update x 0 with d x0 dt = dx dt -σ(t)ϵ. This decomposition allows f (θ t ) to represent the noiseless x 0 (t) instead of x(t), substantially improving the conditioning of J. Fig. 2 illustrates how this separation works in practice.Efficient optimization The Jacobian J in Eq. 2 represents the derivative of the image with respect to the diffrep parameters. For all but the smallest examples, explicitly forming J ⊤ J is computationally intractable. While iterative linear solvers together with modern automatic differentiation frameworks (e.g., [Potapczynski et al., 2024] ) could solve Eq. 2, we found it faster to compute the parameter update as the solution to a non-linear optimization problem. This approach avoids the costly JVP (Jacobian vector product) required to multiply with J ⊤ J.For each step with size ∆t, we solve:EQUATION)As ∆t → 0, using the linearization f (θ -∆θ) = f (θ) -J∆θ + O(∥∆θ∥ 2 ), this optimization objective approaches the linear least-squares solution. The optimal ∆θ ∆t converges to the exact dθ dt from Eq. 2. In practice, we employ the Adam optimizer to solve Eq. 3 for whatever discrete ∆t is used in the diffusion sampling process. This procedure avoids the explicit formation of the linear least-squares solution.Thus far, we have only considered the case where the output of the render function is a single image x ∈ X . However, our interest lies in render functions f that map parameters θ ∈ Θ to an entire view-dependent image space H ⊆ X Π . Using the inner product defined in subsection 2.1, we derive the view-dependent pullback PF-ODE:EQUATIONwhere the expectations are taken over the views π ∼ Π. Here, ∇ log p t (x(π), π) represents the view-specific score function. We use the original score function ∇ log p t (x(π)), with additional view information in the prompt. ϵ(π) denotes the separately managed noise rendered from view π.Examining the optimization form of this equation provides further insight. We can interpret it as minimizing the norm in the function space ∥ • ∥ 2 H = E π∼Π ∥ • ∥ 2 : This is directly analogous to Eq. 3, but with a different inner product. Explicitly written:EQUATIONwhich we can empirically minimize using view samples π ∼ Π.Our method successfully "pulls back" the PF-ODE dx dt using the pull-back of the score function ∇ log p t (x(t)). However, our true goal is to pull back ∇ log p t (x(t)| x 0 (s) ∈ range(f )) for all s ≤ t. This ensures that f can render the noiseless components x 0 throughout the remaining reverse process. For example, when pulling back the PF-ODE for different views of a 3D scene, we want to ensure consistency across all the views. When f is sufficiently expressive and invertible, ∇ logp t (x(t)| x 0 (s) ∈ range(f )) ≈ ∇ log p t (x(t)).This would allow us first to sample x(0) using the PF-ODE from Eq. 1 and then invert f to find θ(0). However, for most significant applications of our method, f is not invertible.In noninvertible cases, consider the sampling trajectory x(t) in X when x 0 (t) / ∈ range f , particularly when no nearby sample has high probability in range f . Each step of Eq. 2 follows the direction in Θ that best approximates the direction of the score model in a least-squares sense. When f is not invertible, J is not full rank, and the update to θ will not precisely match the trajectory in x. Consequently, ∥f (θ t -∆θ) -x(t -∆t)∥ will be significant. The score model, unaware of this, will continue to point towards high-probability regions outside the range of f . For example, consider an f that only allows low-frequency Fourier modes in generated samples. The unconstrained score model might favor images with high-frequency content (e.g., hair, explosions, detailed textures), resulting in blurry, unresolved images. However, suppose the score model was aware of this constraint against high-frequency details. In that case, it might guide samples toward more suitable content, such as landscapes, slow waves, or impressionist art-dominated by expressible low-frequency components. This issue becomes more pronounced when sampling multiple coupled images through f , such as with panoramas and NeRFs, where various views must be consistent.To address this challenge, we adapt the RePaint method [Lugmayr et al., 2022] , initially designed for inpainting, to guide our PF-ODE towards more "renderable" samples. RePaint utilizes the complete Langevin SDE diffusion process, interspersing forward and reverse steps in the sampling schedule. This approach allows the process to correct for inconsistencies during sampling. The forward steps harmonize samples at the current time step by carrying information from earlier steps that implicitly encode the constraint. RePaint requires a stochastic process for conditioning, so we employ the DDIM [Song et al., 2022] sampling procedure with controllable noise at each step. We derive the forward updates of DDIM in Appendix D.Our method introduces significant advancements for sampling diffreps using pretrained diffusion models by performing actual sampling instead of merely mode-seeking, thus capturing the full diversity of high-dimensional distributions. We derive the correct pullback of the PF-ODE, incorporating the essential (J ⊤ J) -1 term to ensure unbiased sampling. Efficiency is enhanced through separated noise handling and a faster suboptimization process, allowing practical application for complex representations. Additionally, our approach extends to coupled images and stochastic functions with implicit constraints, making it suitable for tasks like panorama generation and 3D rendering. A comprehensive summary of the contributions in this section and the pseudocode for the DDRep algorithm using DDIM sampling with RePaint is presented in Appendix E.  
experiments: 
 All our experiments used the Hugging Face implementation of Stable Diffusion 1.5 (SDv1.5) from Rombach et al. [2022] with the default hyperparameters as the noise predictor model. We conducted our experiments on a single NVIDIA A6000 GPU. We used the complete Langevin SDE given by the DDIM [Song et al., 2022] procedure for all our experiments, with η = 0.75 as the stochastic interpolation hyperparameter. We interspersed forward and reverse steps to harmonize the diffrep constraints in the sampling procedure. For the suboptimization problem described in Eq. 3, we used the Adam optimizer for 200 steps.We compare the results of our method with those of SJC [Wang et al., 2022] when used to produce SIRENs [Vincent, 2011] for images. While we could directly fit the SIREN to the final image x(0)-obtained by first solving the PF-ODE in the image space-we use this scenario to compare different training-free generation methods quantitatively.The SIREN we used was a 3-layer MLP mapping the 2D (x, y) ∈ [0, 1] 2 pixel coordinates to the R 4 latent space of SDv1.5. The MLP had a 128-dimensional sinusoidal embedding layer, a depth of 3, a width of 256, and used sin(•) as the activation function. To render an image, we used a 64 × 64 grid of equally spaced pixel coordinates from [0, 0] to [1, 1] with a shape of (64, 64, 2). The MLP RePaint CFG Scale 10.0Figure 5 : Comparison of landscape panoramas sampled using our method. The top panorama is sampled using the RePaint method, while the bottom is sampled without RePaint. Both approaches use 460 function evaluations (NFEs) to ensure fairness and a CFG scale of 10.0. The prompt for these panoramas was "Landscape picture of a mountain range in the background with an empty plain in the foreground 50mm f/1.8".output was a latent image with shape (64, 64, 4), which we decoded using the SDv1.5 VQ-VAE into an RGB image with shape (512, 512, 3). We calculated the final metrics on these rendered images.We used the first 100 captions (ordered by id) from the MS-COCO 2017 validation dataset [Lin et al.] as prompts. We generated three sets of images: (1) reference images using SDv1.5, (2) images rendered from SIRENs sampled using our method, and (3) images rendered from SIRENs generated using SJC [Wang et al., 2022] . We compared these sets against 100 baseline images generated using SDv1.5 with the same prompts but a different seed. We repeated this experiment at five CFG scales [Ho and Salimans] (0, 3, 10, 30, 100). We used the KID metric from Bińkowski et al. to compare the generated images, as it converges within 100 image samples while maintaining good comparison ability. For runtime metrics, see Appendix F.The example images in Fig. 3 (left) demonstrate that the samples generated by our method are almost indistinguishable from the reference. The results in Fig. 3 (right) clearly show that our method produces images on par with SDv1.5 reference samples.To further support this observation, we measured the PSNR, SSIM, and LPIPS scores of our method's samples against the SDv1.5 reference images (Table 1 ). The results show that our method produces SIREN images nearly identical to the reference set.To demonstrate that our method achieves sampling instead of mode-seeking, we plot eight samples using the same prompt for all models in Fig. 4 with different seeds at CFG scale 10. The samples generated by SJC all look very similar, while those generated by our method and SD show significantly more diversity.In this section, we demonstrate how our method can generate SIREN panoramas using the same architecture as the SIREN from subsection 4.1 but with a modified render function. We begin by sampling a view grid of pixel locations with the shape (8, 64, 64, 2) where the batch size is 8, the height and width are 64, and the last 2 dimensions are the (x, y) coordinates. The y values are equally spaced in the range [0,1], and x values are equally spaced in the range [r, r + 1/ar] mod 1.0 for r ∼ U (0, 1). The • mod 1.0 constraint ensures horizontal wrapping, making the panorama continuous over the entire 360 • azimuth.We render the view by sampling from the SIREN at the discrete pixel locations given by the view grid, then decode the latent output using the SDv1.5 VQ-VAE. For our experiments, we assumed that the images were taken by a camera lens with a 45 • field of view (equivalent to a 50mm focal length), corresponding to an aspect ratio of ar = 8.The SIREN panorama imposes implicit constraints to ensure consistency across overlapping views. Fig. 5 and Fig. 7 illustrate the impact of the consistency conditioning described in subsection 3.3. Using RePaint substantially improves the quality and consistency of the generated panorama. This improvement is even more pronounced at low CFG scales, where the diffusion model is encouraged to be more creative and is typically less likely to produce consistent images, as seen in Fig. 7 . For runtime metrics and additional panoramas generated using our approach, see Appendix F.For our NeRF experiments, we employed the voxNeRF architecture used by SJC [Wang et al., 2022] , chosen for its speed and efficiency. While we did not utilize any additional loss terms from their method, we found that incorporating the object-centric scene initialization from Wang et al. [2023] significantly improved our results. This initialization bootstrapped the diffusion process and led to faster convergence. Specifically, we used the initialization function σ init (x) = 10 1 -∥x∥ 0.5 , as detailed in Wang et al. [2023] . For more experimental details, see Appendix F. The NeRFs generated using our method, as illustrated in Fig. 6 , demonstrate our approach's capability to produce detailed, high-quality 3D representations from text prompts. These results showcase the effectiveness of our method in generating complex 3D structures while maintaining consistency across multiple views. 
conclusion: 
 We have presented a comprehensive, training-free approach for pulling back the sampling process of a diffusion model through a generic differentiable function. By formalizing the problem, we have addressed two critical issues with prior approaches: enabling true sampling rather than just mode seeking (crucial at low guidance levels), and addressing a latent consistency constraint using RePaint to improve generation quality. For future work and limitations, see Appendix G.Our method opens up new possibilities for generating complex and constrained outputs using pretrained diffusion models. We hope that future research will build on these insights to further expand the capabilities and applications of diffusion models in areas such as 3D content generation, game design, and beyond. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Wo do not release any data or models in this paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All assets used in the paper are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: It is possible to reparameterize the Gradient Ascent (GA) procedure from Poole et al. [2022] and Wang et al. [2022] in a way that resembles solving the PF-ODE by carefully selecting optimizer hyperparameters and stopping times. However, this reparameterization is highly specific. The crucial difference lies in the interpretation of the trajectory: in GA, the path taken is typically irrelevant as long as it converges to the critical point, allowing us the flexibility to multiply the gradient by any positive semidefinite (PSD) matrix without affecting the solution.For the PF-ODE, however, the trajectory is everything. The stopping time, weighting terms, and score evaluations must be precisely aligned to ensure that the solution looks like a typical sample. Even small deviations in the path, like evolving the PF-ODE for too long or too short, can lead to either oversmoothed samples (with too much entropy removed) or under-resolved, noisy samples. Thus, while GA might superficially look like a discretized version of the PF-ODE, the underlying processes are inherently distinct: one finds the mode, and the other describes the full trajectory required to generate samples from the distribution. In the case of SDS and SJC, they both use Adam as the optimizer, ignoring any of the nuanced considerations needed to transform the PF-ODE into the corresponding optimization procedure.In practice, this distinction means that while both SDS and SJC may yield plausible diffreps at high classifier-free guidance levels, they tend to falter when producing coherent samples under lower CFG weights-particularly for complex, multimodal distributions with many degrees of freedom. High CFG levels may mask these limitations by strongly biasing the output towards the mode, which aligns more closely with the GA approach, but at the cost of reduced sample diversity and potentially missed details captured by the full distribution.The problem with mode-seeking To appreciate the limitations of using the mode as a proxy for sampling, we consider the multi-modal and the high-dimensional settings.In the multimodal setting, the sample distribution may contain several distinct high-density regions. However, mode-seeking algorithms focus on only one of these regions, sacrificing sample diversity.In low-dimensional spaces, the mode often resembles a typical sample. However, this intuition breaks down in high-dimensional spaces (d ≫ 1), such as the space of images. This counterintuitive behavior can be explained using the thin-shell phenomenon. For a high-dimensional standard Gaussian distribution, samples are not concentrated near the mode (the origin), as one might expect. Instead, they predominantly reside in an exponentially thin shell at the boundary of ad-radius ball centered on the mode. This phenomenon explains why the mode can be an anomalous and atypical point in high-dimensional distributions.As an illustrative example, consider sampling a normalized "pure-noise" image. Despite having a mode of 0, we would almost never expect to sample a uniformly gray image. This provides some insight as to why the mode of a distribution with several degrees of freedom will lack the quality and details present only in samples from the thin shell.While RePaint can be used to condition the reverse trajectory, it tends to be less generalizable for several tasks. For more general conditioning-including inverse problems-MCG [Chung et al., 2022] and its extension DPS [Chung et al., 2023] have gained popularity. These methods add a corrective term to the score, encouraging reverse steps to minimize constraint violations for the predicted final sample x0 according to the Tweedie formula. MCG and DPS have seen considerable application and extension to broader settings [Bansal et al., 2023] . Finzi et al. [2023] showed this correction to be an asymptotically exact approximation of ∇ log p t (constraint|x), while Rout et al. [2023] derive the relevant term from Bayes' rule and higher-order corrections.This section provides the details for the derivation of Eq. 2 using differential geometry.First, we provide a full expansion of Eq. 1. For any point in our sample space x ∈ X , we can define a Riemannian metric M x that describes the local geometry of the space around x. Given tangent vectors y, z at point x, we define the inner product using this metric ⟨y, z⟩ Mx = y T M x z where M x is represented using a positive-definite matrix. We implicitly use the Euclidean metric for the image space X , which corresponds to the identity matrix M x = I d , ∀x ∈ X . However, in curved spaces such as the parameter space Θ, the corresponding Riemannian metric is not necessarily the identity and can vary based on where it is evaluated.Here, we use matrix notation rather than Einstein summation or coordinate-free notation for familiarity within the machine learning context. However, we note that matrix notation can obscure some important aspects of differential geometry. For example, while the components of the Euclidean metric in the Euclidean coordinate basis form the identity matrix, the metric is not the identity function, as it transforms between two distinct vector spaces. As such, when using the matrix notation, it becomes very important to keep track of the types of objects rather than just their values.Because we use the Euclidean metric for the image space, its role in Eq. 1 is hidden. Explicitly incorporating the metric into the process, we get the following expanded metric PF-ODEEQUATIONwhere P t is the probability measure at time t and λ x is the Lebesgue measure on X , and dP t /dλ x is the Radon-Nikodym derivative. Notice that when we use M x = I d , ∀x ∈ X , this equation is equivalent to Eq. 1. If we pull back the dynamics of this process through the render function f : X → Θ to the parameter space Θ, we must pull back both the score function and the metric M x to get the pulled back the vector field f * dx dt . The different object types transform as follows with the pullback f * :EQUATION)where J is the Jacobian of f evaluated at θ. For a scalar function s(x), the gradient ∇s(x) is co-vector field (also known as a differential 1-form) and can be converted into a vector field using the inverse metric: M -1 x ∇s(x). We can see that Eq. 10 can be expressed concisely in terms of the pullback metric f * v = (f * M ) -1 f * (M v), which geometrically corresponds to converting the vector field to a covector field with M , pulling back the covector field (using the chain rule), and then converting back to a vector field with the inverse of the pulled back metric. This sequence is illustrated in Fig. 1 . From Eq. 9, the pullback of the Euclidean metric is J ⊤ IJ = J ⊤ J. From these expressions, we can see that the form of f * dx dt should be:f * dx dt = -σ(t)σ(t)(J ⊤ M f (θ) J) -1 J ⊤ M f (θ) M -1 f (θ) ∇ log dP t /dλ x f * dx dt = -σ(t)σ(t)(J ⊤ M f (θ) J) -1 ∇ log dP t /dλ x f * dx dt = -σ(t)σ(t)(J ⊤ J) -1 ∇ log p t (on Euclidean X where M x = I).Thus the pullback of the reverse time PF-ODE isEQUATIONChange of variables contribution. Some readers may be surprised to see that the Jacobian log determinant does not show up in this transformation. Though somewhat technical, this can be seen by unpacking the Radon-Nikodym derivative dP t /dλ x . In general, on a manifold with metric M , the Lebesgue measure is given by dλ = √ det M x dx. Therefore, when written in terms of the density dP t /dx = p t (x), one has dP t /dλ x = p t (x)/ √ det M x . When evaluated only in the Euclidean space where the coordinates are chosen such that M x = I, this detail can be safely ignored. However, when pulling back to the parameter space, these hidden terms matter. The Radon-Nikodym derivative dP t /dλ x is a scalar field and is not transformed under the pullback, however p t is a scalar density and satisfies(f * p t )(θ) = p t (f (θ)) det(J ⊤ J),the familiar change of variables formula. When the input and output dimensions are the same p Θ (θ) = p X (f (θ)) det J. However, √ det M x is also a scalar density and similarly transforms:f * det M x = det f * M x = det J ⊤ M x J = √ det J ⊤ J.Assembling these two components together, we see that the determinant contribution from the change of variables formula cancels with the contribution from the change of the Lebesgue measure in the sampling ODE.f * ∇ log(dP t /dλ x ) = f * ∇ log p t / det M x ) f * ∇ log(dP t /dλ x ) = J ⊤ ∇ log(f * p t ) -log f * det M x f * ∇ log(dP t /dλ x ) = J ⊤ ∇ log p t + 1 2 log det J ⊤ J - 1 2 log det J ⊤ M x J f * ∇ log(dP t /dλ x ) = J ⊤ ∇ log p t .This is why the change of variables term 1 2 log det J ⊤ J term does not appear in the pullback of the PF-ODE.For our experiments, we used the stochastic non-Markovian reverse process from DDIM Song et al. [2022] . For the RePaint steps, we also need to take forward steps to encourage the samples that are "renderable". In this section, we prove the correct form for the forward process looks likeEQUATIONWhere σ(t) is the standard deviation and can be calculated using σ(t) = 1 √ 1/α 2 (t)+1and s(t) is the scale of signal. It can be calculated using s(t) = 1 √ α 2 (t)+1where α(t) is the noise-to-signal ratio. In the main body of the text, we let σ(t) notate the noise-to-signal ratio, and we assume s(t) = 1 but that is not necessarily true in the general case.Theorem D.1. Eq. 12 is the correct forward update for the non-Markovian process.Proof. We will use the notationx(t|0) = x(t) = x(t)|x(0), x(t|t -dt, 0) = x(t)|x(t -dt), x(0), and x(t -dt|t, 0) = x(t -dt)|x(t)x(0).The same holds if we replace x with ϵ 0 .Using the reparametrization trick we can writex(t) = x(t|0) = s(t)x(0) + σ(t)ϵ 0 (t|0), so ln p(x(t|0)) = - 1 2 ϵ 2 0 (t|0) + c.Using DDIM and the reparameterization trick, we can also writeln p(x(t -dt|t, 0)) = - 1 2 1 τ (t) x(t -dt) - s(t -dt) τ (t) x(0) - σ 2 (t -dt) -τ 2 (t) τ (t) ϵ 0 (t) 2 + c = - 1 2 σ(t -dt) τ (t) ϵ 0 (t -dt) - σ 2 (t -dt) -τ 2 (t) τ (t) ϵ 0 (t) 2 + cNow we can use Bayes' theoremp(x(t|t -dt, 0)) = p(x(t -dt|t, 0))p(x(t|0)) p(x(t -dt|0)) ln p(x(t|t -dt, 0)) = - 1 2 σ(t -dt) τ (t) ϵ 0 (t -dt) - σ 2 (t -dt) -τ 2 (t) τ (t) ϵ 0 (t) 2 + ϵ 2 0 (t) -ϵ 2 0 (t -dt) + c = - 1 2 σ(t -dt) τ (t) ϵ 0 (t) - σ 2 (t -dt) -τ 2 (t) τ (t) ϵ 0 (t -dt) 2 + c = - 1 2 • σ 2 (t -dt) τ 2 (t) ϵ 0 (t) - σ 2 (t -dt) -τ 2 (t) σ(t -dt) ϵ 0 (t -dt) 2 + cUsing reparameterization, we can say that (conditioned on x(0)), x(t|t -dt, 0) = s(t)x(0) + σ(t)ϵ 0 (t|t -dt, 0), whereϵ 0 (t|t -dt, 0) ∼ N σ 2 (t -dt) -τ 2 (t) σ(t -dt) ϵ 0 (t -dt), τ 2 (t) σ 2 (t -dt)Our method introduces several crucial innovations for sampling differentiable representations using pretrained diffusion models:1. True sampling vs. mode-seeking: Unlike SJC and SDS, which primarily find modes of the distribution, our method aims to generate true samples. This distinction is critical for capturing the full diversity and characteristics of the underlying distribution, especially in high-dimensional spaces. 2. Correct pullback of the PF-ODE: We derive the mathematically correct form of the pulled-back Probability Flow ODE, which includes a crucial (J ⊤ J) -1 term. While this term acts as a preconditioner in optimization-based approaches (affecting only convergence rates), it is essential for unbiased sampling in our PF-ODE framework. Omitting this term leads to incorrect results. 3. Efficient implementation: We introduce techniques for efficient implementation, including separated noise handling and a faster suboptimization procedure. These innovations allow for the practical application of our method to sample complex differentiable representations. 4. Generalization to coupled images and stochastic functions: Our method extends naturally to scenarios involving coupled images and stochastic functions, making it applicable to a wide range of problems, including 3D rendering and panorama generation. 5. Handling multimodality and constraints: Our sampling approach naturally handles multimodal distributions, unlike mode-seeking methods. However, this introduces challenges when the samples are incompatible with the constraints imposed by the differentiable representation. We address this by conditioning the sampling process with the consistency constraint using an adapted RePaint method.These key ideas collectively enable our method to generate high-quality, diverse samples of differentiable representations by encouraging the diffusion model to maintain consistency with the constraints imposed by the representation.The pseudocode for the DDRep algorithm using DDIM sampling with RePaint is presented in Alg. 1.This section presents additional details and runtime metrics for our algorithm with the experimental settings given in section 4. σ langevin = η σ -2 (t) + 1 1 -σ 2 (t-∆t)+1σ 2 (t)+16:ϵ langevin ∼ N (0, σ 2 langevin )7:Sample Views: π ∼ Π 8:if reverse then 9:x(t) = f (θ, π) + σ(t)ϵ(π)10:εt (π) = ε(x(t), t, view_prompt(π), cfg)11:x next = x(t) -σ(t)ε t (π) + 1 -σ 2 langevin σ(t -∆t)ε t (π) + σ(t -∆t)ϵ langevin 12:else 13: 1 : Image similarity scores (mean ± std) for the images produced using our method and those produced using the SD reference.x next = f (θ, π) + σ(t) 1 -σ 2 langevin ϵ(π) + σ(t)Image SIRENs Sampling a batch of eight reference images using SDv1.5 takes 39 seconds.Generating SIRENs using SJC depends on the number of iterations used; running SJC for 3000 iterations to generate eight SIRENs takes 694 seconds, corresponding to a per-iteration time of 2.31 seconds. In comparison, sampling a batch of eight SIRENs using our method takes 82 seconds. These timings demonstrate that our method offers a significant speed advantage over SJC while producing comparable results to the reference SDv1.5 samples.The PSNR, SSIM, and LPIPS scores of our method against the SDv1.5 reference images are presented in Table 1 .SIREN Panoramas Generating a SIREN panorama with our method takes 218 seconds. Fig. 7 shows how using RePaint to condition the sampling procedure can generate more coherent panoramas for low CFG scales. Additional examples of SIREN panoramas generated using our method are provided in Fig. 8 .Our algorithm processed a batch of eight views per iteration, which was the maximum capacity for the A6000 VRAM. Our work has demonstrated the zero-shot generation of implicit images, panoramas, and NeRFs. The versatility of our approach opens up a wide range of potential applications, including vector graphics generation, embedding diverse content in scenes through geometric transformations, differentiable heightmaps, and applications using differentiable physics renderers.While we focused on NeRFs in this paper, more advanced differentiable representations like Gaussian splats and full-scene representations have emerged. Our method should be applicable to these cases, but further work is needed to adapt our approach to these newer representations.Looking ahead, one could envision generating entire maps or game environments using diffusion models. However, this would likely require innovations in amortization and strategies for splitting and combining subproblems. In these cases, the function f may be even more restrictive than for SIRENs and NeRFs, underscoring the importance of conditionally sampling the implicit constraint.Limitations Our method faces two significant limitations. First, the additional steps required for RePaint introduce computational overhead. A priori, we cannot determine how many steps are needed to harmonize the constraint into the diffrep. Transitioning to a conditional sampling method like MCG could potentially allow us to skip these extra forward steps and directly integrate the constraint into the solver. However, this requires further investigation to ensure compatibility with our method.Second, the substantial stochasticity in the Monte Carlo estimate of the pullback score over several views, particularly when the Jacobian of the render function is ill-conditioned, poses a challenge. While increasing the number of sample views could reduce variance in our estimate, it would also slow down our algorithm and consume more memory. Further research into view sampling techniques, such as importance sampling, could help decrease variance without compromising computational efficiency.• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: Our contribution is methodological and we don't know how it may be used. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).38th Conference on Neural Information Processing Systems (NeurIPS 2024).Note that E[ϵ] = 0, so the additional -ϵ term in the SDS objective behaves as an unbiased variance reduction term for the score estimate since Cov(ε, ϵ) > 0.One may be tempted to impose that the metric is Euclidean both in the X space and the Θ space, but these two choices are incompatible because the render transformation maps between the two spaces. 


Decision: This paper proposes a zero-shot method for sampling implicit representations from pretrained diffusion models, which could in principle allow for better coverage of the full distribution, whereas prior methods might be more susceptible to mode collapse. The reviewers generally found the paper interesting and the contribution promising, but identified limited scope of experiments (for example the need for comparison to more baselines, and to validate the diversity claims in further settings) as a weakness, as well as some issues with clarity or errors in presentation. After discussion, some of these concerns were addressed — particularly presentational ones — but the concerns about the coverage of the experiments remain. Overall, the conceptual contribution seems potentially valuable and impactful, but it would need a more convincing and comprehensive demonstration of the benefits to achieve optimal impact.

Reviews: review: paper_summary: paper_summary: The present paper introduces a training-free method to sample differentiable representations using pre-trained diffusion models. This is achieved by pulling back the dynamics of the reverse-time process from the image to the parameter space. Moreover, training-free methods for conditional sampling are employed to (approximately) satisfy implicit constraints. Numerical experiments show the performance of the method for images, panoramas, and 3D NeRFs. In particular, it is shown that the method can improve upon the baselines in terms of quality and diversity of generated representations.
strength_weakness: strength_weakness: 1. Baselines and related works: Further discussion of related works and additional numerical comparisons are needed.
    - It would be helpful to provide further discussion on related works, such as Zero-1-to-3, HiFA, Magic123, LatentNeRF, Fantasia3D, ...
    - While the presented results are promising, additional experiments and visualizations would strengthen the validation of the method.
    - A more detailed comparison to methods requiring fine-tuning/training could provide a clearer picture of the method's performance. In particular, it would be good to add comparisons in terms of (overall) runtime/NFEs.
    - Why are there no PSNR/SSIM/LPIPS numbers in Table 1 for the baselines, e.g., SJC?
    - While DreamFusion is similar to SJC, it would still be interesting to add empirical comparisons.

2. Presentation and theory: Further details and explanations could be provided (e.g., also in the appendix).
    - The geometric perspective and pullback operation might be challenging for readers not well-versed in these areas. One could improve accessibility by providing more intuitive explanations.
    - Precise details on how RePaint is adapted to this setting seem to be missing. In this context it would also be good to clarify the connections to other methods for posterior sampling (DPS/ReSample/...).
    - It would be helpful to add some explanation on the "separately managed noise object" $\epsilon(t)$ or $\epsilon(\pi)$.
questions: questions: Why is the score not the gradient of a scalar function?

Typos:
1. The visualizations in Fig. 1 (right) seem not to match the explanation "The example images in Figure 1 (right) show that the samples produced by our method are almost indistinguishable from the reference."
2. The order (top/bottom) in the caption of Fig. 4 is wrong.
limitations: limitations: Only a few limitations are mentioned and it would be good to enumerate potential failure cases of the proposed method.
paper_summary: paper_summary: This paper introduces a novel, training-free method to sample through differentiable functions using pretrained diffusion models.
strength_weakness: strength_weakness: More systematic evaluation of the method in addition to image examples would be great.
questions: questions: I do not have questions at this stage.
limitations: limitations: The authors have discusses some limitations.
paper_summary: paper_summary: The paper introduces a method for sampling a differential representation using a pre-trained diffusion model. Instead of sampling in the image space, the authors propose sampling in the parameter space of differential representations by 'pulling back' the probability distribution from the image space to the parameter space of the differential representation.
strength_weakness: strength_weakness: - The technical part of the paper is difficult to follow.
- Comparison to the state-of-the-art is provided only in Section 5.1, where a differential representation is not required.
- The code is not shared at this stage.
questions: questions: - The two main relevant papers (Poole et al. [2022] and Wang et al. [2022]) are not discussed in appropriate detail:
  - The authors claim equivalence between the methods without providing a detailed argument or a citation where the argument is made.
  - Lines 93-94 suggest that these methods act as "mode-finding algorithms," however, a discussion is lacking.

- The arguments in Section 3.1 (which I believe is the main argument of the paper) are hard to follow for a non-expert:
  - What is the relevance of Equations 3-5?
  - How are Equations (6) and (7) derived?
limitations: limitations: yes
paper_summary: paper_summary: This paper introduces a zero-shot method to sample neural representations with pre-trained diffusion models. By pulling back the measure over the data space through the representation, the authors express the PF ODE in the parameter space. Solving this ODE can directly provide parameter samples of the representation. The authors also discuss cases where the representation needs to model coupled images. In this scenario, the PF ODE is expressed by expectation and can be solved following the same principle. This method offers random samples rather than returning the mode, which can yield results with higher diversity compared to baselines without sacrificing performance. The method is also compatible with RePaint, which can ensure the implicit constraint (e.g., the inductive bias of the representation or practical consistency constraint).
strength_weakness: strength_weakness: My primary concern is its runtime. If I understand correctly, you need to run the optimization for several iterations for each data point and each Euler step. And for tasks like NeRF, where the PF ODE involves expectations, this optimization also involves an MC estimator. However, this paper only reported the runtime for 3D NeRF experiments. I am curious on 

a. what is the runtime for other experiments? What is the runtime for the baselines? Is this method significantly more expensive than others?

b. How many samples do you use to estimate the expectation? Does it contribute to the runtime? Will reducing the number of samples increase the variance and lead to suboptimal performance?

I would still appreciate this approach even though it may be more expensive. However, reporting the runtime and comparing it with the previous baselines can help us understand this method better.
questions: questions: I have two more questions regarding the experiments. 
1. In Fig 1 (right), you compare your method with SJC and claim that "the samples produced by our method are almost indistinguishable from the reference." However, I do not necessarily agree with this claim. Your method yields much smoother images than SD. Even though I mainly suspect this may be due to SIREN's inductive bias, it is visible and distinguishable. Additionally, it seems that your method provides very different outcomes with CFG 30, while results by CFG 3/10/100 are pretty similar. Do you have any explanation for this? Is it a sign that this approach is not robust?
2. The PF ODEs in all experiments need to meet some implicit constraints due to the inherent bias of INRs. However, you only apply RePaint for Panorama SIRENs to ensure the consistency of the generated panorama. Do I understand this correctly? Is there a difference if you ensure the implicit constraints for other experiments? Is it possible to compare the results with and without ensuring the constraint?


In general, I appreciate the contribution of this work and would be happy to raise my score if my questions are adequately addressed.
limitations: limitations: I did not find discussions on limitations even though Sec 6 is called Limitations and Conclusion. I am unsure if this work has no limitations, such as runtime. Could the author please clarify this?

Comments: comment: I thank the authors for their rebuttal and explanations and raised my score. I understand the difference between mode finding and sampling, however, I still think that this should be further validated experimentally, e.g., further visualizations like Fig. 2 (also for 3D NeRFs) as well as comparison to other baselines (even though they are *conceptually* similar to SDS/SJC). In that spirit, I thank the authors for providing additional metrics but agree that other metrics should be considered that can compare distributions instead of instances.
I feel my concern has already been addressed. Therefore, I raise my rating for this work.
Thank you for your comment.

In our experiments, we used 100 inference (reverse, Euler) steps of DDIM and a single forward step for RePaint at every increment except the first one. This schedule corresponds to 199 steps in total. We optimize the INR using 200 Adam steps for each inference step.

Generating a batch of 8 reference images takes 39 seconds. Generating 8 SIREN images using SJC depends on the number of iterations used. Running SJC for 3000 iterations to generate 8 SIRENs takes 694 seconds, corresponding to a per-iteration time of 2.31 seconds. Generating a non-SIREN panorama with eight views using our method takes around 41 seconds, which makes sense since this is essentially the same problem as generating eight batched images.

We highlight some specific limitations from Section 6 and include some additional ones here:

- One significant limitation of our method is the additional steps we need for RePaint. A priori, we do not know how many steps are required to harmonize the constraint into the diffrep. Moving to a conditional sampling method like MCG would allow us to skip the extra forward steps and directly integrate the constraint into the solver. However, this needs further exploration to work with the rest of our method.
- There is also often significant stochasticity in the Monte Carlo estimate of the pullback score over several views, especially when the Jacobian of the render function is ill-conditioned. We would prefer to take more sample views to reduce the variance in our estimate, but this slows down our algorithm or takes more memory. More work is needed to improve the view sampling techniques (like importance sampling) to decrease the variance in our estimates without slowing down our algorithm.
- Finally, we only investigated our method on NeRFs; since then, there have been significantly more advanced diffreps like Gaussian Splats and full-scene representations. While our method should also work for these cases, more work needs to be done to delineate the specifics of how to adapt our method for these cases.

We will add all these details to the final version of the paper.
Thank you for your reply and for providing more details. However, I have 3 more further questions:


1. How many iterations per Eular step do you use for optimization? 

2. >  Generating eight batched SIREN images using our method takes 82 seconds. Generating a SIREN Panorama with our method takes 218 seconds.

What is the baseline methods' runtime? 

2. My concern regarding limitations has not been addressed yet. Did you discuss limitations in Section 6? Or do you think there is no clear limitation?