Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision
Authors: Ayush Tewari
Tianwei Yin
George Cazenavette
Semon Rezchikov
Joshua B. Tenenbaum
Fredo Durand
William T. Freeman
Vincent Sitzmann
Venue: NeurIPS 2023 spotlight (2023)
Keywords: 3D generative models
neural rendering
neural scene representations
NeRF
diffusion models
differentiable rendering
inverse graphics
inverse problems
TL;DR: We propose a novel class of diffusion models that learn to sample from distributions of signals that are never directly observed, e.g., our model learns to sample 3D scenes by only training on 2D image observations.

Abstract: Denoising diffusion models are a powerful type of generative models used to capture complex distributions of real-world signals. However, their applicability is limited to scenarios where training samples are readily available, which is not always the case in real-world applications. For example, in inverse graphics, the goal is to generate samples from a distribution of 3D scenes that align with a given image, but ground-truth 3D scenes are unavailable and only 2D images are accessible. To address this limitation, we propose a novel class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never directly observed. Instead, these signals are measured indirectly through a known differentiable forward model, which produces partial observations of the unknown signal. Our approach involves integrating the forward model directly into the denoising process. A key contribution of our work is the integration of a differentiable forward model into the denoising process. This integration effectively connects the generative modeling of observations with the generative modeling of the underlying signals, allowing for end-to-end training of a conditional generative model over signals. During inference, our approach enables sampling from the distribution of underlying signals that are consistent with a given partial observation. We demonstrate the effectiveness of our method on three challenging computer vision tasks. For instance, in the context of inverse graphics, our model enables direct sampling from the distribution of 3D scenes that align with a single 2D input image.

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision
Ayush Tewari'
Tianwei Yin'
George Cazenavette'
Semon Rezchikov*
Joshua B. Tenenbaum 122.
Frédo Durand'
William T. Freeman'
Vincent Sitzmann'
MIT CSAIL
MIT BCS
MIT CBMM
*Princeton IAS
Abstract
Denoising diffuuion models have emerged as a powerful class of generative models capable of capturing the distributions of complex, real-world signals. However, current approaches can only model distributions for which training samples are directly accessible, which not the case in many real-world tasks. In inverse graphics, for instance, we seek to sample from a distribution over 3D scenes consistent with an image do not have access to ground-truth 3D scenes, only 2D images. We present a new class of conditional denoising diffusion probabilistic models that learn to sample from distributions of signals that are never observed directly, but instead are only measured through known differentiable forward model that generates partial observations of the unknown signal. To accomplish this, directly integrate the forward model into the denoising process. At test time, our approach enables us to sample from the distribution over underlying signals consistent with some partial observation. We demonstrate the efficacy of our approach on three challenging computer vision tasks. For instance, inverse graphics, demonstrate that model in combination with 3D-structured conditioning method enables us to directly sample from the distribution of 3D scenes consistent with single 2D input image.
Introduction
Consider the problem of reconstructing 3D scene from a single picture. Since much of the 3D scene is unobserved, there are an infinite number of 3D scenes that could have produced the image, due to the 3D-to-2D projection, occlusion, and limited field-of-view that leaves a large part of the 3D scene unobserved. Given the ill poo-posesssss of this problem, it is desirable for a reconstruction algorithm to be able to sample from the distribution over all plausible 3D scenes that are consistent with the 2D image, generating unseen parts in plausible manners. Previous data-completion methods, such as in-painting in 2D images, are trained on large sets of ground-truth output images along with their incomplete (input) counterparts. Such techniques do not easily extend to 3D scene completion, since curating large dataset of ground-truth 3D scene representations very challenging.
This 3D scene completion problem, known as inverse graphics, is just one instance of a broad class of problems often referred as Stochastic Inverse Problems, which arise across scientific disciplines whenever we capture partial observations of the world through a sensor. In this paper, we introduce a diffusion-ber based framework that can tackle this problem class, enabling us sample from a distribution of signals that are consistent with a set of partial observations that are generated from the signal by con-invertible, generally nonlinear, forward model. For instance, in inverse graphics, we learn to sample 3D scenes given an image, yet never observe paired observations of images and 3D scenes training time, nor observe scenes directly.
Equal Contribution. Project page: diffusion-wid wow.oloo-rtttt--oooddd..oeee models w......ub.......................
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While progress in deep learning for generative modeling has been impressive, this problem remains unsolved. In particular, variational autoencoders and conditional neural processes are natural ap- proaches but have empirically fallen short of modeling the multi-modal distributions required in, for instance, inverse graphics. They have so far been limited to simple datasets. Emerging diffusion models [1], in contrast, enable sampling from highly complex conditional distributions but require samples from the output distribution that is to be modeled for training, e.g. full 3D models. Some recent work in inverse graphics has resorted to two-stage approach, where one first reconstructs a large dataset of 3D scenes to then train an image-conditional diffusion model to sample from the conditional distribution over these scenes [2, 3]. To avoid a two-stage approach, another recent line of work trains a conditional diffusion model to sample from the distribution over novel views of scene, only requiring image observations at training time [4, 5]. However, such methods do nor model the distribution over 3D scenes directly and therefore cannot sample from the distribution over 3D scenes consistent with an image observation. Thus, multi-view consistent 3D scene can only be obtained in a costly post-procsssing stage [6]. A notable exception is the recently proposed RenderDiffusion [7], demonstrating that it is possible to train an unconditional diffusion model over 3D scenes from observing only monocular images, While one can perform conditional sampling even with unconditional models, they are fundamentally limited simple distributions, in this case, single objects in canonical orientations.
Our core contribution is novel approach for integrating any differentiable forward model that describes how partial observations are obtained from signals, such as 2D image observations and 3D scenes, with conditional denoising diffusion models. By sampling an observation from our model, we jointly sample the signal that gave rise to that observation. Our approach has a number of advantages that make it highly attractive for solving complex Stochastic Inverse Problems. First, our model is trained end-to-end and does away with two-stage approaches that first require reconstruction of a large dataset of signals. Second, our model directly yields diverse samples of the signal of interest. For instance, in the inverse graphics setting, our model directly yields highly diverse samples of 3D scenes consistent with an observation that can then be rendered from novel views with guaranteed multi-view consistency. Finally. our model naturally leverages domain knowledge in the form of known forward models, such as differentiable rendering, with all guarantees that such forward models provide. We validate our approach on three challenging computer vision tasks: inverse graphics (the focus of this paper), as well as single-image motion prediction and GAN inversion.
In summary, we make the following contributions:
1. We propose a new method that integrates differentiable forward models with conditional diffusion models, replacing prior two-step approaches with conditional generative model trained end-to-end.
2. We apply our framework to build the first conditional diffusioon model that learns to sample from the distribution of 3D scenes trained only on 2D images. In contrast to prior work, we directly learn image-conditional 3D radiance field generation, instead of sampling from the distribution of novel views conditioned on context view. Our treatment of inverse graphics exceeds mere application of the proposed framework, contributing a novel, 3D-structured denoising step that leverages differentiable rendering both for conditioning and for the differentiable forward model.
3. We formally prove that under natural assumptions, as the number of observations of each signal in the training set goes to infinity, the proposed model maximizes not only the likelihood of observations, but also the likelihood of the unobserved signals.
4. We demonstrate the efficacy of our model for two more downstream tasks with structured forward models: single-image motion prediction, where the forward model is a warping operation, and GAN inversion, where the forward model is a pretrained StyleGAN [8] generator.
2 Method
Consider observations (0), (i)) that are generated from underlying signals Sj according to a known forward model forward(). i.e., O2 = forwarddSS,, o5), where of are parameters of the forward model corresponding to observation O Each observation can be purtial. Specifically, given single observation, there is an infinite number of signals that could have generated this observation. However, we assume that given a hypothetical set of all possible observations, the signal is fully
2

---

(a) Data: Tuples of observations (O,o)
(c) Graphical Model of Diffusion Process
forward dd)
Oo
T 0 fre
C
C 0 Octx
observed
unobserved
forward
P/S.
(b) Single Denoising Step
ST
S
Sc1
So
(Tai TT.T)
denoise
forward
, tig
TT
T
Tca
To
Figure I: Overview of our proposed method. (a) We assume a dataset of tuples of observations (0, generated from unobserved signals S via differentiable forward model. (b) We propose to integrate the forward model directly into the denoising step of diffusion model: given pair of observations of the same signal, we designate context O Odhi and target O tMr We add noise O 055 then feed (Ost 2 O, & to a neural network denoise to estimate the signal S=-1. We then apply the forward model to obtain an estimate of the clean larget observation, tg (c) The graphical model the diffusion process.
determined. In the case of inverse graphics, Oj are image observations of 3D scenes S and of are the camera parameters, where we index scenes with j and observations of the j-th scene via i. forward() is the rendering function. Note that if we were to capture every possible image of a 3D scene, the 3D scene uniquely determined, but given single image, there are an infinite number of 3D scenes that could have generated that image, both due to the fact that rendering is projection from 3D and 2D, and due to the fact that a single image only constrains the visible part of the 3D scene. We will drop the subscript j in the following, and leave it implied that always consider many observations generated from many signals. Fig. provides an illustration of the data.
We are now interested in training a model that, at test time, allows us to sample from the distribution of signals that are consistent with previously unseen observation O. Formally, we aim to model the conditional distribution OS|O, o). We make the following assumptions:
We have access differentiable implementation of forward(). We have access to a large dataset of observations and corresponding parameters of the forward model, {(O', Φ²) N.
In our training set, we have access to several observations per signal.
Crucially, we do not assume that we have direct access the underlying signal that gave rise particular observation, i.e., we do not assume access to tuples of (O, db S). Further, we also do not assume that we have access any prior distribution over the signal of interest, i.e., a dataset of signals of the form {S) J,. and thus cannot train a generative model to sample from an unconditional distribution over signals.
observe
Recent advances in deep-learning-bare generative modeling have seen the emergence of denoising diffusion models powerful generative models that can be trained to generate highly diverse samples from complex, multi-mode distributions. We are thus motivated to leverage denoising diffusion probabilistic models to model P(S/O, o). However, existing approaches cannot be trained if we do not have access to signals S. In the following, we give background on denoising diffusion models and discuss the limitation.
2.1 Background: Denoising Diffusion Probabilistic Models and their Limitation
Denoising diffusion probablistic models are a class of generative models that learn to sample from distribution by learning to iteratively denoise samples. Consider the problem of modeling the distribution pe (x) over samples x. A forward Markovian process g(Xur)) adds noise the data as
q(x² xx-1) = Nxxx /1 ,,
(1)
Here, 9., E ....T are the hyperparameters that control the variance schedule. A denoising diffusion model learns the reverse process, where samples from a distribution p(ær) = N( I) are transformed incrementally into the data manifold as px(xT.)) = p(ZT) Pe pxxx111 x:), where
(x xx) = -N f), t)).
(2)
neural network denoiseø() with learnable parameters learns to reverse the diffusion process. It is also possible to model conditional distributions ( c), where the output is computed as
3

---

denoiseg(X,, t, c). The forward process does not change in this case; in practice, we merely add the conditional signal input the denoising model.
Unfortunately, we cannot train existing denoising diffusion models to sample from p(S | (0,0) fact, even from unconditional distribution p(S). This would require computation of the Markovian forward process in Eq. 1. However, recall that we do not have access to any signals {S²}; Jy we thus can not add any noise any signals to then train a denoising neural network. In other words, since no S is directly observed, we cannot compute qS, S)
Φ), or, in
2.2 Integrating Denoising Diffusion with Differentiable Forward Models
We now introduce class of denoising diffusion models that we train to directly model the distribution p(S Ock; zostt) over signals S given an observation (Oett dea)) Our key contribution to directly integrate the differentiable forward model forward() into the iterative conditional denoising process. This enables us to add noise to and denoise the observations, while nevertheless sampling the underlying signal that generates that observation.
Our model is trained on pairs of "context" and "target" observations of the same signal, denoted O and Ourrr, As in conventional diffusion models, for the forward process, we have q (0ttt ug t
O D ) OO VI on Oct:
3 11 B.I). In the reverse process, we similarly denoise Ourt conditional
Po O
ugt O:7
ctat
jug)
p( het II Po (O ngt
0
tggt O detti
However, unlike conventional diffusion models, we implement po O fngt first predicting an estimate of the underlying signal Su11 and then mapping it to an estimate of the denoised observations via the diffcrentiable forward:
1he O d (feet) by
t-1
S denoise, (Ouu O t, dett duet).
(4)
trys f Forward d
(5)
E-1
irgi
NOOO tral O 8-1 11.1)
(6)
Here, Óugl is an estimate of the clean observation, and the constants Ct.1 and 3r. are chosen to match the total noise added by the forward process at time 1-1. See Fig. 1 for overview. At test time, signal is sampled by iterating Eq. 4, 5, and 6 starting with p( _ ( (O, Importantly, our models define a generative model over the underlying signal via Eq. 4:
T
Pow (SaT
ctxt,
II
Po (S.1
ugg
ret
We will suppress the subscript in the notation, and refer to this distribution as p(So:r for brevity from now
O detxt)
Loss Function. We train to minimize the following two loss terms:
Ch
Egua our dem ,pur, | forvard ettxl 0 tt, deta (0t3),, ) ) ||2
6
(8)
Caveel E Exs Quand gea aore gmorel
| novel forward(denoisse (O O it, it, g dug) σ novel
[|2]
Here, compute the estimate of the observation from the target, well as separate, novel forward model parameter gnovel, In the supplemental document, we show that these losses approximate a total observation loss, maximizing the likelihood of all possible observations of the signal S.
Characterizing the Conditional Distribution Over Signals. Due to the complexity of the reverse process, it may not be clear that the learned distribution over signals will agree with the true distribution, even in the limit of infinite data. However, this model will indeed asymptotically learn the true conditional distribution over signals, as we formally prove in the supplement:

---

i Render Deterministic Conditioning Input
Generate 3D Scene & Target View
Ruumnuurg zumph
Generated 3D Soene S
PIIN
NLP
plrgt
MLP.
Generated
Trgt. View O
Target pose fx
Encoder enc
Noisy target 0
Encoder
Encoder enc.-u
Ctxt. image O pose gf
Figure 2: Overview of 3D Generative Modeling, We build 3D-structured denoise operator top of pixeINeRF [9] that learns to sample from the distribution of seenes from image observations only. Given context image Oan with camera pose oth we pick a target pose कtLt We render out a deterministic estimate of the depth, RGB, and features of the target view det using pixel-aligned features f extracted from the context view with encoder enc;=0 (left, only RGB shown here). To generate 3D scene, concatenate the deterministic estimate with noise O, and extract features f. for the targer view with enc,- f and fos now jointly parameterize the radiance field the generated scene S.-1, and we may render an estimate of the clean larget view O The model is trained end-to-end via re-rendering loss.
Proposition 1. Suppose that any signal S can be reconstructed from the set of all all possible observations of S. Under this assumption, if in the limit as the number of known observations per signal goes to infinity, there are parameters such that C + _novel is minimized, then the conditional probability distribution over signals discovered by our model p(S Octit df agrees with the true distribution ptrue(S Oux, djet)).
The proof follows by showing that our losses implicitly minimize a diffusion model loss over total observations, which are collections of all possible observations of our signal. As such, when the observations suffice completely reconstruct the signal, the correctness of the estimated distribution over total observations forces the estimated distribution over signals to be correct, as well.
3 Prior Work on Latent Variable Models for Inverse Problems
Variational Autoencoders [10, 11], normalizing flows [12], conditional [13] and attentive neural processes [14] are latent-variable models that can be combined with forward models learn sample from the distribution of unobserved signals from observations [15, 16]. However, they empirically fall short of accurately modeling complex signal distributions in inverse graphies, for instance, such models have so far been limited to synthetic 3D scenes. Generative Adversarial Networks can be trained with differentiable forward models in-the-loop, and have yielded impressive results in unconditional generative modeling of unobserved signals [17-19]. Similarly, in concurrent work, diffusion models have been leveraged for unconditional generative modeling through differentiable forward models [2, 7, 20]. However, unconditional models are limited to tight distributions, and no conditional generative modeling of similar quality has been demonstrated. Diffusion models trained directly on signals have been effectively applied to diverse inverse problems such super- resolution [21-25], inpainting [21, 23-26], and medical imaging [27]. These works utilize the learned prior of the data distribution recover the latent signal through a "plug and play" approach [28-30], integrating the diffusion model with forward measurement process according to Bayes' rule. These approaches versatile and can easily adapt new inverse problems without retraining. However, unlike our models, they rely on direct supervision over the signals in the form large datasets.
4 Applications
We now apply our framework to three stochastic inverse problems. focus on applications in computer vision, where we tackle the problems of inverse graphics, single-image motion prediction, and GAN inversion. For each application, we give detailed description of the forward model, the dataset and baselines, well as brief description of prior work.
5

---

Context Frame
Target Frame
Target Depth
Novel Intermediate Views Frame
Intermediate Depths
Frame 50
Frame
Frame 50
Sample 2 Sanple
GT
Detern
Sumple 3
Context
Sample 2 Sample
GT
Detet
Figure Sample Diversity. We illustrate different 3D scenes sampled from the same context image for RealEstate 10k and Co3D datasets. Unlike deterministic methods like pixcINeRF [9], our method generates diverse and distinct 3D scenes that all align with the context image. Co3D results generated using autoregres- sive sampling. where 360 degree trajectory photorealism and diversity of generated structures note the high-fidelity geometry of the occluded parts of the hydrant and the diverse background appearance.
generated by iteratively sampling target images. Note the
indoor scene, such as doors cabinets. Also
4.1 Inverse Graphics
We seek learn a model that, given single image of a 3D scene enables us to sample from the distribution over 3D scenes that are consistent with the observation. We expect that 3D regions visible in the image are reconstructed faithfully, while unobserved parts are generated plausibly. Every time we sample, we expect a different plausible 3D generation. Signals S are 3D scenes, and observations 2D images O and their camera parameters At training time, assume that have access to at least two image observations and their camera parameters per scene, such that we can assemble tuples of (Oetxt t Ongt d with 2D images Octxt Orgt and camera parameters de g
Scope. We note that our treatment of inverse graphics exceeds mere application of the presented framework. In particular, we not only integrate the differentiable rendering f forward function, but further propose novel 3D-structured denoise function. Here, we enable state-of-the-arr conditional generation of complex, real-world 3D scenes.
Few-shot reconstruction of 3D scene representations via differentiable rendering was pioneered by deterministic methods [9, 31, 32, 32-41] that blur regions of the 3D scene unobserved in the context observations. Probabilistic methods have been proposed that can sample from the distribution of novel views trained only on images [4, 5, 42-45J. While results are impressive, these methods do not allow sampling from the distribution of 3D scenes, but only from the distribution of novel views. Generations are not multi- consistent. Obtaining 3D scene requires costly post-processing via score distillation [6]. Several approaches [2. 3] use two-stage design: they first reconstruct dataset of 3D scenes, and then train 3D diffusion model. However, pre-computing large datasets is expensive. Further, to obtain high-quality results, dense observations are required per scene. RenderDiffusion [7] and HoloDiffusion [20] integrate differentiable forward rendering with an unconditional diffusion model. enabling unconditional sampling of simple, single-object scenes. Similar to us, RenderDiffusson performs denoising in the image space, while HoloDiffusion uses 3D denoising architecture. Other methods use priors learned by text-conditioned image diffusion models to optimize 3D scenes [46-48]. Here, the generative model does not have explicit knowledge about the 3D information of scenes. These methods often suffer from geometric artifacts.
Related Work.
Structure of S and forward model render. We can afford only an abridged discussion here please see the supplement for more detailed description. We use NeRF [49] as parameterization of 3D scenes, such that S is function that maps 3D coordinate p to a color and density as S()) (a, c). We require generalizable NeRF that is predicted in feed-forward pass by an encoder that takes set of M context images and corresponding camera poses {(O,, J. input.

---

Input
PixeINeRF
SparseFusion
Ours
Input pixeINeRI
SparseFusion
Ours
Diffusion Out. Distill.
Diffusion
Distill.
GT
GT
No Depth
No Depth
Input
Input
GT
GT
No Depth
No Depth
Figure 4: Qualitative Comparison for Inverse Graphies application. We benchmark with SparseFusion [S] deterministic pixelNeRR [9]. SparseFusion samples novel views conditioned on deterministic rendering (Diffusion Out.), and generates multi-view consistent 3D scenes only after Score Distillation. Our method consistently generates higher-quality scenes, while directly sampling 3D scenes.
We base our model on pixelNeRF [9]. pixelNeRF first extracts image features {F,}; from each context observation via an encoder enc as F, = enc(O:). Given 3D point p, it obtains pixel coordinates each context view via p, = (p, Ø,) via the projection operator я, and recovers a corresponding feature as f = F.(,,")) by sampling the feature map at pixel coordinate p." It then parameterizes S via an MLP as:
S(p) = (a(p), = MLP(((f, + M.
(10)
where is concatenation and p, is the 3D point p transformed into the camera coordinates of observation i. The number of context images M is flexible, and we may condition S on single or several observations. It will be convenient refer pixelNeRF that is reconstructed from context and target observations (Oext graa)) and (Orrg ((t) as
S(- eenc (OtM) enc 1ryt
(11)
where we make the pixelNeRF encoder enc explicit and drop the poses हरT and daNt We leverage differentiable volume rendering [49] forward model, such that
render(S... Φ),
(12)
where S is rendered from camera with parameters Φ.
Implementation of denoise. Fig. gives an overview of the denoising procedure. Following our framework, we obtain the denoised target observation
trgt
L-1 as:
rret
render ( where
(13)
S = S( onc;=o(00M)) enc
(14)
where the image encoder enc; is conditioned on the timestep f. other words, will generate target view Ô 6-1 by rendering the pixelNeRF conditioned on the context and noisy target observations. However, feeding the noisy 0.r directly to pixelNeRF is insufficient. This is because the pixel- aligned features (nc,(O) are obtained from each view separately thus, the features generated by enc, (ett)) will be uninformative. To successfully generate 3D scene, we have to augment the O with information from the context view. We propose to generate conditioning information for O by rendering a deterministic estimate O pixelNeRF only on the context view, and render an estimate of the target view via volume rendering. However, in the extreme case of completely uncertain target view, this results in completely blurry image. We thus propose to additionally render high-dimensional features. Recall that any 3D point p. have (o(p), c(p)) (L), We modify MLP; to also output digh-dimensional feature and
des render (S(-) enc,=o (Ott))) "n2t). Le., we condition

---

Single-Image Motion Prediction
Probabilistic GAN Inversion
Input
Motion Field Samples
Determ.
Patch
Sumples from GAN inversion
Detern
Magnitude
Magnitude
Figure Qualitative Results for Single- Image Motion Prediction (left) and GAN Inversion (right).
render deterministic feature map augment O (only RGB shown in figure). We generate the final 3D scene as S S(- ence=o (Octor enc, Oee O)² The final denoised target view is then obtained according to the rendering Eq. 13 above.
Loss and Training. Our loss consists of simple least-squares terms re-rendered views, identical to the general loss terms presented in Eqs. 8 and 9, addition to regularizers that penalize degenerate 3D scenes. We discuss these regularizers, as well as training details, in the supplement.
4.1.1 Results
Datasets We evaluate on two challenging real-world datasets. We use Co3D hydrants [50] to evaluate our method on object-centric scenes. For scene-level 3D synthesis, RealEstate 10k dataset [51], consisting of indoor and outdoor videos of scenes.
the challenging
Baselines We compare our approach with state-of-the-aa approaches in deterministic and probabilistic 3D scene completion. We use pixeINeRF as the representative method for deterministic methods that takes single image as input and deterministically reconstructs a 3D scene. Our method is the first to probabilistically reconstruct 3D scenes in an end-to-end manner. Regardless, we compare with the concurrent SparseFusion [52] that learns image-space generative model over novel views of a 3D scene. Score distillation of this generative model is required every time we want to obtain a multi-view consistent 3D scene, which is costly.
Qualitative Results. In Fig. 3, we show multiple samples of 3D scenes sampled from a monocular image. For the indoor scenes of RealEstate 10k, there are large regions of uncertainty. We can sample from the distribution of valid 3D scenes, resulting in significantly different 3D scenes with plausible geometry and colors. The objects are faithfully reconstructed for the object-centric Co3D scenes, and the uncertainty in the scene is captured. We can sample larger 3D scenes and render longer trajectories by autoregressive sampling, i.e., we treat intermediate diffused images as additional context observations to sample another target observation. The Co3D results in Fig. were generated autoregressively for complete 360 degrees trajectory. In Fig. 4, we compare our results with pixelNeRF [9] and SparseFusion [5]. pixelNeRF a deterministic method and thus leads to very blurry results in uncertain regions. SparseFusion reconstructs scenes by score-distillation over a 2D generative model. This optimization is very expensive, and does not lead to natural-looking results.
Quantitative Results. For the object-centric Co3D dataset, we evaluate the accuracy of novel views using PSNR and LPIPS [53] metrics. Note that PSNR/LPIPS are not meaningful metrics for large scenes since the predictions have a large amount of uncertainty. i.e., a wide range of novel view images can be consistent with any input image. Thus, we report FID [54] and KID [55] scores evaluate the realism of reconstructions in these cases. Our approach outperforms all baselines for LPIPS, FID, and KID metrics, as our model achieves more realistic results. We achieve slightly lower PSNR compared to PixeINeRF [9]. Note that PSNR favors mean estimates, and that we only evaluate our model using a single randomly sampled scene for an input image due to computational constraints.
4.2 Simgle-Image Motion Prediction
Here, we seek to train model that, given single static image, allows us to sample from all possible motions of pixels in the image. Given, for instance, an image of person performing a task, such as kicking a soccer ball, it is possible to predict potential future states. This is a stochastic problem, as there are multiple possible motions consistent with an image. We train on a dataset of natural videos [56]. We only observe RGB frames and never directly observe the underlying motion, i.e, the pixel correspondences in time are unavailable. We use tuples of two frames from videos within a small temporal window, and use them context and target observations for training.
8

---

3D Scene Completion
GAN Inversion
Co3D
RtalEstate 10k
FFHQ
PSNRT
LPIPS
FIDL
KIDI
FIDJ
KIDI
FID,
KIDJ
PixeINeRF
17.93
0.54
180.20
0.14
195.40
0.14
Determ.
25.7
0.019
SparseFusion
12,06
0.63
252.13
0.16
99.44
0.04
Ours
7.45
0.002
Ours
17.47
0.42
84.63
0.05
42.84
0.01
Table 1: Quantitative evaluation. (left) We benchmark our 3D generative model with state-of tate--ofthh-arr baselines
pixelNeRF [9] and SparseFusion [5]. (right) We benchmark with deterministic baseline on GAN inversion, which drastically outperform.
Related Work. Several papers tackle this problem, where motion in the form of optical flow [57-59], 2D trajectories [60, 61], and human motion [62, 63] are recovered from a static image; however, all these methods assume supervision over the underlying motion. Learning to reason about motion requires the neural network to learn about the properties and behavior of the different objects in the world. Thus, this serves as useful proxy task for representation learning, and can be used backbone for many downstream applications [60, 64].
Structure of I and forward model warp. Our signal S stores the appearance and motion information 2D grid. At any pixel u, the signal is defined S(u) (S. Sm (u)), where Su(u) R is the color value, and Sm (u) E I2 is 2D motion vector. The forward model is a warping operator, such that warp(S, Φ) (u a + e (u)) (u) and ø a scalar that changes the magnitude of motion. We implement this function using differentiable point splatting operation [65].
Implementation of denoise. The inset figure illustrates our design. We use 2D network that takes Ockit 0 and t as input, and generates the motion map S., as the output. The signal is then reconstructed as S (O S.). Context and target frames correspond to parameters df = 0 and क = 1, and can be reconstructed from the signal using warp
Frame Noise Ques 0
Motion S.i
Frame O
deno Lare
Loss and Evaluation. Similar to inverse graphics, we use reconstruction and regularization losses. The reconstruction losses are identical to Eqs. 8 and 9, and the regularization loss is smoothness term that encourages natural motion of the scene, see supplement for details. We show results in Fig. 5 (left), where we can estimate a diverse set of possible motion flows from monocular images. By smoothly interpolating o, we can generate short video sequences, even though our model only saw low- www-framrrte video frames during training. We also train deterministic bascline, which only generates a single motion field. Due to the amount of uncertainty in this problem, the deterministic estimate collapses to near-zero motion field regardless of the input image, and thus, fails to learn any meaningful features from images.
4.3 GAN Inversion
Projecting images onto the latent space of generative adversarial networks is well-studied problem [8, 66), and enables interesting applications, as manipulating latents along known directions allows user to effectively edit images [67-69]. Here, we solve the problem of projecting partial images: given a small visible patch in an image, our goal is to model the distribution of possible StyleGAN2 [8] latents that agree with the input patch. There correspond to the input observation, and we train our method without observing supervised (image, latent) pairs. Instead, train on pairs of (Oeta O Ogg observations, where Ottxt are the small patches in images, and ugg are the full images.
diverse set of latents that can
Related Work. While most GAN inversion methods focus on inverting complete image into the generator's latent space [70-78], some also reconstruct GAN latents from small patches via supervised training. Inversion is not trivial, and papers often rely regularization [77] or integrate the inversion with editing tasks [79] for higher quality. We also integrate the inpainting task with the inversion, and seek to model the uncertainty of the GAN inversion task given only partial observation (patch) of the target image.
Structure of S and forward model synthesize. Our signal S E RR512 is 512 dimensional latent code representing the "w" space StyleGAN2 [8] trained on the FFHQ [80] dataset. The forward model synthesize(S, Φ) [SN(S) first reconstructs the image corresponding to using

---

forward pass of the GAN. It then extracts a patch using the forward model's parameters ø that encode
the patch coordinates.
Patch
Noise
t
Latent
Image
Implementation of denoise, Loss, and Evaluation. Please see
O o
Sut
O
the inset figure for illustration of the method. The denoising network receives Oati O, and timestep as input, and generates an estimate of the StyleGAN latent w. The loss function is identical to Eq. 8 and compares the reconstructed sample with ground truth. We show results in Fig. 5 (right). We obtain diverse samples that are all consistent with the input patch. We also compare with deterministic baseline that minimizes the same loss but only produces a single estimate. While this deterministic estimate also agrees with the input image, it does not model the diversity of outputs. We consequently achieve significantly better FID [54] and KID [55] scores than the deterministic baseline, reported in Tab. I (right).
5 Discussion
Limitations. While our method makes significant advances generative modeling, it has several limitations. Sampling 3D scenes at test time can be very slow, due to the expensive nature of the denoising process and the cost of volume rendering. We need multi-view observations of training scenes for the inverse graphics application. Our models are not trained on very large-scale datasets, and can thus not generalize to out-of-distribution data.
Conclusion We have introduced a new method that tightly integrates differentiable forward models and conditional diffusion models. Our model learns to sample from the distribution of signals trained only using their observations. We demonstrate the efficacy of our approach on three challenging computer vision problems. In inverse graphics, our method, in combination with 3DD-structured conditioning method, enables us to directly sample from the distribution of real-world 3D scenes consistent with a single image observation. We can then render multi-view consistent novel views while obtaining diverse samples of 3D geometry and appearance in unobserved regions of the scene. We further tackle single-image conditional motion synthesis, where we learn to sample from the distribution of 2D motion conditioned sample images that exist in the latent space of GAN that are consistent with a given patch. With this work, make contributions that broaden the applicability of state-of-the thethhr-ttt generative modeling to a large range of scientifically relevant applications, and hope to inspire future research in this direction.
single image, as well as GAN inversion, where we learn to
Acknowledgements. This work was supported by the National Science Foundation under Grant No. 221 1 1259, by the Singapore DSTA under DST000EC120300823 (New Representations for Vision), by the NSF award 1955864 (Occlusion and Directional Resolution in Computational Imaging), by the ONR MURI grant N00014-22-1-27400 and by the Amazon Science Hub. We are grateful for helpful conversations with members of the Scene Representation Group David Charatan, Cameron Smith, and Boyuan Chen. We thank Zhizhuo Zhou for thoughtful discussions about the SparseFusion baseline. This article solely reflects the opinions and conclusions of its authors and no other entity.
Author contributions. Ayush and Vincent conceived the idea of diffusion with forward models, designed experiments, generated most figures, and wrote most of the paper. Ayush contributed the key insight to integrate differentiable rendering with diffusion models by denoising in image space while generating 3D scenes. Ayush and Vincent generalized this to general forward models, and conceived the single-image motion application. Vincent contributed the 3D-structured conditioning and generated the overview and methods figures. Ayush wrote all initial code and ran all initial experiments. Ayush and Tianwei implemented the inverse graphics application and generated most of the 3D results of our model, while George helped with the baseline 3D results. Ayush executed all single-image motion experiments. George conceived, implemented, and executed all GAN inversion experiments. Semon helped formalizing the method and wrote the proposition and its proof. Frédo and Bill were involved in regular meetings and gave valuable feedback on results and experiments. Josh provided intriguing cognitive science perspectives and feedback on results and experiments, and provided significant part of the compute. Vincent's Scene Representation Group provided significant part of the compute, and the project profited from code infrastructure developed by and conversations with other members of the Scene Representation Group.
10

---

References
[1] Jascha using nonequilibrium thermodynamics. Pruc. ICML. 2015.
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning
[2] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guiie guided 3d radiance field diffusion. Proc. CVPR, 2023. 2, 5,
Scung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, and Sanja Fidler. Neuralficld-Idms Scene generation with hicrarchical latent diffusion models. Proc. CVPR, 2023. 2,6 6
[4] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. arXiv preprint arXiv:2334.02600 2023. 2,
[5] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffussio for 3d recon- struction. Proc. CVPR, 2023. 2, 6,7.8.9
[6] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. Proc. ICLR, 2023. 2
[7] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusionn Image diffusion for 3d reconstruction, inpainting and generation. Proc. CVPR, 2023. 2, 5,6 6
[8] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proc. CVPR, 2020. 2, 9
[9] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proc. CVPR, 2021. 5, 6, 7, 8,
[10] Diederik Kingma and Max Welling. Auto-encoding variational bayes. Proc. ICLR, 2014.
[11] Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In Proc. JCML, 2014.
[12] Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. Proc. ICML, 2015.
[13] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In Proc. ICML, 2018.
[14] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. Proc. ICLR, 2019. 5
[15] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In Proc. ICML, 2021. S
[16] Pol Moreno, Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Rosalia G Schneider, Björ Winckler, Larisa Markeeva, Théophane Weher, and Danilo Rezende. Laser: Latent representations for 3d generative modeling. arXiv preprint arXiv:2301.057477 2023.
[17] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021. 5
[18] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin. Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Proc. NewrIPS, 2022.
[19] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. In Proc. ICCV, 2021. 5
[20] Animesh Karewwa, Andrea Vedaldi, David Novomn, and Niloy Mitra. Holodiffusion: Training a 3d diffuuson model using 2d images. Proc. CVPR, 2023. 5, 6
[21] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In Proc. ICLR, 2023. 5.
11

---

[22] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusiio probabilistic models. Proc. ICCV, 2021. 5
[23] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proc. NeurIPS, 2022. 5
[24] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverss--uuide diffusion models for inverse problems. In Proc. ICLR, 2023. 5
[25] Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Advances in Neural Information Processing Systems, 1--11322-132555 2021. 5
[26] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. ICLR, 2021. 5
[27] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems medical imaging with
score- score-based generative models. Proc. ICLR, 2022. 5
[28] Johnathan Bardsley. Meme-based image reconstruction with uncertainty quantification. SIAM Journal on Scientific Computing, 34(3):A 1316--133333 2012. 5
[29] Singanallur Venkatakrishnam, Charles A. Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. 2013 IEEE Global Conference on Signal and Information Processing, pages 945 9988,
2013. 5
[30] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4):1804-1844, 2017.
[31] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Proc. NeurIPS, 2019.
[32] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020. 6
[33] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalow, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos the wild. In Proc. CVPR,
2021.
[34] Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural seene representations from a single image. In Proc. ICLR. 6
[35] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proc. ICCV, 2021. 6
[36] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning render novel views from wide-baseline stereo pairs. In Proc. CVPR, 2023.
[37] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proc. ICCV, 2021.
[38] Mohammed Suhail. Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In Proc. ECCV, 2022. 6
[39] Julian Chibane. Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proc. CVPR, 2021. 6
[40] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proc. CVPR, 2021. 6
Ricardo
[41] Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. Coconets: Continuous contrastive 3d seene representations. In Proc. CVPR, 2021. 6
[42] Daniel Watson, William Chan, Ricardo Martin- Marain-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. Proc. ICLR, 2023. 6
[43] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gareelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394)) 1204 1214-121001111111111111001101110 2018. 6
12

---

[44] Hung- Yu Tseng. Qinbo Li, Changil Kim, Suhib Alsisan. Jia-Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. arXiv preprint urXiv:2303.17594 2023.
[45] Jiatao Gu, Qingzhe Gao, Shuangfei Zhai, Baoquan Chen, Lingjie Liu, and Josh Susskind. Learning controllable 3d diffusion models from single-view images. arXiv preprint arXiv:2304.06700, 2023. 6
[46] Luke Melas-Kyriazi Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360 (ddeg) reconstruction of any object from a single image. Proc. CVPR, 2023. 6
[47] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias NieBner. Text2room: Extracting textured meshes from 2d text-to-image models. arXiv preprint arXiv:2303.1195: 2023.
[48] Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01 133, 2023. 6
[49] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020. 6, 7
[50] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proc. ICCV, 2021. 8
[51] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Truns. Graph. (Proc. SIGGRAPHI) 37, 2018. 8
[52] Yi Ding, Alex Rich, Mason Wang, Noah Stier, Matthew Turk, Pradeep Sen, and and Tobias Höllerer. Sparse fusion for multimodal transformers. arXiv preprint arXiv:21 11.22111111999... 2021. 8
[53] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metrie. In Proc. CVPR, 2018. 8
[54] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Proc. NeurlPS, 2017. 8, 10
[55] Mikolaj Binkkwski,, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 001,0, 2018. 8, 10
[56] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127:1 1125, 2019. 8
[57] Ruohan Gao, Bo Xiong, and Kristen Grauman. Im2ftow: Motion hallucination from static images for action recognition. In Proc. CVPR, 2018.
[58] Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical flow prediction from static image. In Proc. ICCV, 2015. 9
[59] Silvia Pintea, Jan C van Gemert, and Amold WM Smeulders. Déja vu: Motion prediction in static images. In Proc. F.CVV, 2014. 9
[60] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autocncoders. In Proc. ECCV. Springer, 2016. 9
[61] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lai, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In Proc. ICCV, 2018. 9
[62] Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Vidco forecasting by generating pose futures. In Proc. ICCV, 2017. 9
[63] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In Proc. ECCV, 2018. 9
[64] Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Guess what moves: unsupervised video and image segmentation by anticipating motion. arXiv preprint arXiv:2205.07844, 2022. 9
[65] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proc. CVPR, 2020. 9
[66] Jun Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipulation on the natural image manifold. In Proc. ECCV, 2016. 9
13

---

[67] Erik Härkönen, Aaron Hertzmannn Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Proc. NeurIPS, 2020.
[68] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Berardd, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer. and Christian Theobalt. Stylerig: Rigging stylegan for 3d control portrait images. In Proc. CVPR, 2020.
[69] Yujun Shen, Ceyuan Yang. Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence. 44(4):2004 2018, 2020.
[70] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proc. ICCV, 2019.
[71] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit embedded images? In Proc. CVPR, 2020. 9
[72] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Yan Zhu, Antonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020. 9
[73] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based stylegan encoder via iterative refinement. In Proc. ICCV, 2021.
[74] Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang. Collaborative learning for faster stylegan embedding. arXiv preprint arXiv:2007.01758 2020. 9
[75] Stanislay Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proc. CVPR, 2020. 9
[76] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In Proc. CVPR, 2021. 9
[77] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021. 9
[78] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. In Proc. CVPR, 2022. 9
[79] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theohalt. Pie: Portrait image embedding for semantic control. ACM Transactions on Graphics (TOG), 39(6): 14, 2020. 9
[80] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. CVPR, 2019.
14


Reviews:
[{"review_id": "FRIiLolY66", "reviewer": "Reviewer_5usH", "summary": "This paper focuses on denoising diffusion models, a type of generative model used to capture complex signal distributions. However, current approaches can only model distributions when training samples are available, which is not always the case in real-world tasks. In fields like inverse graphics, where the goal is to sample from a distribution of 3D scenes based on a given image (without having access to ground-truth 3D scenes), this limitation poses a challenge.\n\nTo address this, the authors introduce a new class of denoising diffusion probabilistic models. These models learn to sample from distributions of signals that are never directly observed but instead measured through a known differentiable forward model. This forward model generates partial observations of the unknown signal. The authors integrate this forward model directly into the denoising process.\n\nDuring testing, this approach enables sampling from the distribution of underlying signals consistent with a given partial observation. The authors demonstrate the effectiveness of their method on three challenging computer vision tasks. For instance, in inverse graphics, they show that their model allows direct sampling from the distribution of 3D scenes consistent with a single 2D input image.", "questions": "1. Though this method gives a theoretical formulation of this problem, to some extent I find it quite similar to the underlying idea of RenderDiffusion (CVPR 23'), which also trains a diffusion model which each denoising step output the denoised X0.  Though Renderdiffusion does not leverage the trgt-ctxt pair for training, the \"learning 3D from 2D projection\" spirit looks similar. I would like to hear the author's comments on this.\n\n2. I wonder if your method share the same setting with \"Generative novel view synthesis with 3d-aware diffusion models.\" which trains an independent diffusion model on a particular scene (like Hydrant and a single scene from RealEstate10k? This is not a limitation but since EG3D/RenderDiffusion like method could perform well on a single category, e.g., FFHQ/AFHQ/Shapenet, is your method also capable of achieving that performance, though they do unconditional generation and your method is a conditional one.\n\n3. Since currently GT 3D poses are required, I wonder is this method robust to the noisy pose problem? Besides, since your incorporate the volume rendering as the \"forward\" process, is the synthesized scene perfect view consistent? \n\n4. I do appreciate your experiment on RealEstate10K, and I wonder whether your method could be generalized to the \"scene-level\" 3D generation such as urban / BlockNeRF-like setting? If not, what is the challenge.\n\n5. What are the limitations / failure cases of this method and could you share some insight behind it?\n\n6. Regarding the GAN inversion experiment, I wonder whether your method has any intrinsic advantages over 3D GAN inversion, since you used StyleGAN2 in the experiment while this method is designed for 3D. Comparison with \"E3DGE: Self-Supervised Geometry-Aware Encoder for Style-based 3D GAN Inversion, CVPR 23\" is welcome since you both adopt encoder-based pipeline.\n\n7. I see that your method achieves much better performance against SparseFusion. I wonder the popular SDS loss is still helpful in your framework and do you see it as a necessary component in the future 3D generation task? Thanks for your elaboration. ", "limitations": "Though some concerns, overall I find this paper a sound submission and I hold my current rating towards accept. I am looking forward to the authors' elaboration in the rebuttal.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper proposes a new type of conditional diffusion model for 3D scene generation without referring to the underlying 3D signal. This is an essential problem in the present era. The writing is super clear and I can easily follow. The experiments is sound and demonstrate the effectiveness of the proposed method. Moreover, I do appreciate the in-depth analysis and formulation of the \"3D generation w/o 3D data\" challenge.", "weaknesses": "Though this method gives a theoretical formulation of this problem, I believe more elaboration on some details and comparisons with some baselines are required to address my concerns. I have left my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. The reviewer nicely summarizes our paper and notes that the “writing is super clear”, we solve “an essential problem in the present era”, and that our experiments “demonstrate the effectiveness...”. We now address the points raised in the review. \n\n### Related Work\n**RenderDiffusion** :  See general comment (G1)\n\n**Generative novel view synthesis with 3d-aware diffusion models (GeNVS)**: Please note that GeNVS is concurrent to our submission. Both our method and GeNVS train on a dataset of scenes, and not a single scene. However, there are fundamental differences. While we integrate the forward model with the denoising diffusion architecture to enable 3D scene generation, GeNVS is limited to only generating 2D images from the diffusion model. This is a crucial difference: Genvs is *not* a 3D generative model, i.e. it is not capable of generating 3D scenes. Rather, it can only generate *novel views* of 3D scenes, and thus would require score distillation to obtain a 3D scene, a limitation that GeNVS shares with SparseFusion, which we benchmark with.\n\n**EG3D**: EG3D is a 3D GAN that is trained from 2D images. It belongs to a class of 3D GANs that are primarily limited to unconditional modeling of simple objects (as the reviewer correctly notes). Our approach, for the first time, enables learning 3D diffusion models of complicated scenes, such as RealEstate10k - note that 3D GANs have never been able to demonstrate generative modeling of scenes of comparable complexity. We have not explored training on monocular image collections in our paper. Monocular images make the learning problem under-constrained (because of the lack of ctxt-trgt pairs), and exploring how our theoretical formulation can be extended to such cases is a very interesting problem that we leave for future exploration. \n\n### Robustness to noisy poses\nWe agree with the reviewer that our method requires GT 3D poses. We performed an experiment where we added noise to the pose parameters. The results gracefully degrade - training with 5% noise added to the camera translation parameters makes the results blurry, and leads to worse performance quantitatively, see Fig. 1 and Tab. 1 in the rebuttal pdf. Relying on GT poses is a very common requirement in contemporary 3D reconstruction methods, and we will mention this in our limitations. Removing this reliance on known accurate poses is an important problem for future research. \n\n### View consistency\n\nVolume rendering can indeed lead to imperfect view consistency, depending on the quality of training supervision. As we train on videos where we have reasonably dense supervision, we found our results to be highly 3D consistent. To demonstrate this, we visualize point clouds extracted from the reconstructed 3D volume from extreme out-of-distribution viewpoints, see Fig. 5 in the rebuttal pdf. These point clouds demonstrate that our method extracts reasonable surfaces in 3D. \n\n### Scene-level generation\nWe thank the reviewer for their suggestion. We show results on trajectories that enable moving from one room to another in the case of RealEstate10k, and moving all around an object in the case of Co3D. Extending to city-scale trajectories would require tackling similar challenges as BlockNeRF, where the method would need to forget context images that are further in the past, and be able to merge several independent NeRFs. The main limitation at this point is the compute resources required to train and sample longer trajectories. We believe that these limitations are transitory and we would see large scale scene synthesis in the near future, with more optimized architectures and better hardware. \n\n### Limitations\nSee general comment (G4)\n\n### 3D GAN inversion\nOur 2D GAN experiment showed that our framework is capable of using non-linear GAN-based forward models, and can help with inference from partial data. 3D GAN inversion is interesting as, similar to our inverse graphics task, it also involves tackling the uncertainties that arise from projection, while constraining the reconstructions to the GAN manifold. Our framework should provide the same advantages it currently does over the deterministic baselines on inverse graphics. Nevertheless, we aim to conduct this experiment for the revised version of our paper. \n\n### SparseFusion and SDS\nThank you for the insightful question! Our paper shows that SDS is not **necessary** for 3D generation, as we demonstrate that it is possible to learn a 3D generative model without SDS. This resolves some of the limitations that come with SDS: it has mode-seeking behavior, it requires costly test-time optimization, and it usually relies on monocular image priors that lead to 3D artifacts (like the infamous Janus artifacts). While some recent approaches make progress on some of these limitations, we show that we can completely side-step these issues using our novel approach that natively learns 3D models. At the same time, we also believe that SDS can be **complementary** to our approach: we could use SDS as an additional form of supervision at training time to supervise text conditioning, use it at test time to constrain the sampled scenes conditioned on text, or to extract high-fidelity mesh models. We will add a discussion of these exciting topics to our future work section.\n"}, {"review_id": "l0f0hghT7K", "reviewer": "Reviewer_bLEB", "summary": "This paper incorporates a differentiable forward model (e.g., rendering function) into a denoising probabilistic process in order to sample from distributions of underlying signals consistent with partial observations. For example, in inverse graphics the proposed approach would enable sampling from the distribution of 3D scenes consistent with a single 2D image.\n\nThe efficacy of the approach is showcased in three applications: sampling from the distribution of 3D scenes using only 2D images at train and test times; single image motion prediction (where the forward model is a warping operation); and projecting partial images onto the latent space of a GAN (GAN inversion).\n", "questions": "**Q1.** How does your approach relate / differ w.r.t. the recent methods in W1 above?\n\n**Q2.** What are the main limitations of the approach, e.g., complexity of training and inference?", "limitations": "No limitations or negative societal impact were discussed in the main submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** Diffusion models for sampling 3D shapes or scenes have only been explored recently. A key challenge in this direction is the lack of 3D training data. The method proposed in this paper is able to train 3D models without the need for 3D training data.\n\n**S2.** Tackling Stochastic Inverse Problems more generally is a great way to broaden interest in advances in diffusion models.", "weaknesses": "**W1.** There are some (very) recent works that tackle the same problem tackled in this paper, e.g., (Karnewar et al. CVPR 2023), (Kim et al. CVPR 2023). It would be important for the authors to discuss their work in relation to such recent works.\n\n**W2.** The evaluation is limited: few tasks, few baselines (e.g., compare with the more detailed evaluation in baseline SparseFusion [50]). In fact, for single-image motion prediction and GAN inversion the main result is in the form of predictions for two inputs.\n\n**W3.** While the approach was framed as a solution to Stochastic Inverse Problems in general, all applications considered are in computer vision. In my view, the generality of the problem statement is a strength of the submission and a demonstration of this generality would have greatly strengthened the submission.\n\n### References\n\nKarnewar et al. HOLODIFFUSION: Training a 3D Diffusion Model using 2D Images, CVPR 2023.\n\nKim et al. NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models. CVPR 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. The reviewer notes that our paper tackles a “key challenge” in training 3D diffusion models, and that our general formulation is a “great way to broaden interest in advances in diffusion models”. We now address the concerns: \n\n### HoloDiffusion [Karnewar et al.] and NeuralField-LDM [Kim et al.]: \nWe address this remark in the following points: \n- Please note that we already discuss these papers in the original submission (HoloDiffusion in L162-163, NeuralField-LDM in L190) and talk about how they relate to our work. **However, we hear the reviewer in that this discussion may not cover all essential points - we will thus significantly expand on this discussion in the camera-ready submission as below.**.\n- Note that both of these papers are concurrent to our submission and do not constitute prior work.\n- While certainly related and worth discussing, their contributions do not significantly overlap with our contributions and they do not take away from the significance of our results. We discuss this in-depth in the following, **and will extend the discussion of these papers in the revised version of our paper**.\n    - HoloDiffusion is an unconditional 3D generative model limited to simple object-centric 3D scenes. Specifically, it can only be trained on pre-segmented (background-free) Co3D objects, one class at a time. HoloDiffusion adds noise to deterministic estimates of 3D scenes. This input differs at training and at test time, inducing a domain shift, which the authors address by introducing a 2-stage approach. In contrast, our model does not suffer from a domain shift at training and at test time, meaning that we can train our model end-to-end and in a single stage, without any multi-stage pipeline. Our model can further be trained on the much more complex class of indoor rooms from the RealEstate10k dataset, and further can model Co3D objects without pre-processing via segmentation and background removal. We present theoretical insights in our paper that clearly show how our formulation lets us optimize for 3D scenes using our 2D loss functions. As shown in our paper, we can learn distributions over complex compositional scenes that go far beyond the results of HoloDiffusion. \n    - NeuralField-LDM is a 2-stage approach, where 3D latents are first computed from the 2D data, and then a 3D diffusion model is trained on the 3D latents. Our approach is radically different, demonstrating that we can directly learn to sample in 3D by learning from 2D data without requiring computation of an intermediate stage of 3D latents.\n\n\n### Limited Evaluation\nWe believe that evaluations presented in our paper are detailed and demonstrate the effectiveness of the approach. This is acknowledged by 5usH (“experiments… demonstrate the effectiveness of the proposed method”), LcVC (“results (are) comprehensive and solid”), and zgt6 (“Results seem strong, both visually and quantitatively”). We now expand on the points raised by the reviewer. \n\nWe evaluate against the state of the art in both deterministic and probabilistic reconstruction, pixelNeRF and SparseFusion, respectively. The reviewer points out the evaluation in SparseFusion, which features more deterministic baselines. Please note that pixelNeRF outperforms all other deterministic baselines in the SparseFusion evaluations (see their Table 2 PSNR). Thus, we believe that comparing with SparseFusion and pixelNeRF sufficiently demonstrates the quality of our method.\n\nWhile SparseFusion only uses the Co3D dataset for evaluation, we also use the very challenging scene-level RealEstate10k dataset. Nevertheless, we provide even more evaluations of our 3D reconstruction in this rebuttal where we train a general model on 10 categories of the Co3D dataset, see general discussion point (G3) and the rebuttal PDF. \n\n\nFor the single-image motion and GAN inversion experiments, we do provide more results than the two examples that the reviewer mentions:\nWe provide additional results in the supplemental pdf and the webpage.\nFor the GAN inversion experiment, we provide quantitative results in Table 1 demonstrating improvements over a deterministic baseline. We further mention that the deterministic baseline for the single-image motion problem collapses (L290 - 291), thus making any quantitative evaluation meaningless. \nWe are not aware of any other probabilistic baselines that we could compare to. For example, all encoder-based GAN inversion approaches we are aware of use a deterministic model. However, we are happy to add any additional evaluations that the reviewer suggests. \n\n\n### Limitations\nSee general discussion point (G4)\n\n### Applications beyond Computer Vision\nWe believe our framework is very generally applicable, and would be useful for a wide range of applications. However, we agree with the reviewer and will clearly mention the scope of our empirical result in the abstract and introduction of the paper. We address the generality of our approach in the general discussion point (G2), and discuss new experiments and potential applications. \n"}, {"review_id": "va6moHnatq", "reviewer": "Reviewer_LcVC", "summary": "This work proposes a framework that allows learning a conditional diffusion model for a signal without direct observations. The proposed method integrates differentiable forward models with conditional diffusion models. It is evaluated for the task of learning image-conditioned 3D scenes from 2D images with poses and a few more downstream tasks.", "questions": "1. If the context is empty, i.e. making the model unconditional, is proposed method mathematically equivalent to [1] RenderDiffusion? If not, what are the major differences in formulations?\n2. In line 137 equation (4) (5) (6), to my understanding, \\hat{O} is the \"predicted x0\" in DDPM, it is an estimate of clean observation but in theory not an instance in the space of clean observations. \\hat{O} is rendered from S_{t-1}, while a true clean observation has a corresponding neural field, is it a practical fact or something provable that an \"estimated observation\" \\hat{O} has a corresponding neural field S_{t-1}?\n\n[1] RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method addresses an important problem. A main limitation of diffusion models is that they typically require direct observations on the signal of interest. The proposed method showed promising results on learning with indirect observations only.\n2. The proposed method is novel (by considering RenderDiffusion as concurrent), which extends the classical diffusion process. The network architecture is well-designed and intuitive. \n3. The effectiveness of the proposed method is demonstrated on multiple tasks and datasets, making the results comprehensive and solid.", "weaknesses": "I did not find any major weakness in this paper.\nA minor weakness: on CO3D dataset, only hydrants category is evaluated. It could make the results stronger to show the performance on more categories.\nSome other points are put into Questions section, which may make the submission stronger if they are clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful read and insightful comments. The reviewer notes that our method “addresses an important problem” and shows “promising results”, our architecture is “well-designed and intuitive”, and our results are “comprehensive and solid’.  We now address remaining concerns: \n\n### Evaluation on more Co3D categories\n\nThank you for the suggestion. We evaluate the suggested setting, training a single model on all 10 Co3D categories. See general discussion point (G3) for details, and the rebuttal PDF for results. This evaluation goes beyond the category-specific training used in other existing papers, such as SparseFusion, which require training a single model per class. While our model generates plausible results, the diffusion model in SparseFusion often fails to generate reasonable images, even generating output images from a different object category. Score distillation with such a model often fails to reconstruct any reasonable object. We further conducted an experiment where train on a subset of the large-scale object-centric Objaverse dataset. Both these experiments show that our models are scalable and effective. \n\n### Differences to RenderDiffusion\nEven if the context is empty, our model is different from RenderDiffusion, as it does not rely on global pose information, i.e. assuming that all objects are aligned with one canonical coordinate frame, and does not rely on monocular supervision. We rely on the multi-view losses for our proof in the supplemental. See general comment (G1) for a more detailed discussion. \n\n### Does every “estimated observation” (\\hat{O})  have a corresponding signal (S_{t-1})?  \nIf we understand this right, the reviewer raises the point that while the “true” clean observation has a corresponding signal, is it necessarily true that the estimate of this clean observation \\hat{O} also has a corresponding signal? In our models, \\hat{O} is generated using a forward model operation on the estimate of the signal, and thus, the diffusion model is strictly limited to estimating \\hat{O} that have a corresponding signal. If the estimated observation does not have any corresponding signal, we would not be able to properly minimize our loss functions. Thus, it is a practical fact that, for the forward models we describe in the paper, the estimate of the observation can also be described as a signal, letting us successfully optimize our networks. Thanks for the insightful question!"}, {"review_id": "dqh6Bi7ILB", "reviewer": "Reviewer_zgt6", "summary": "The authors present a new method for training diffusion models based on partial observations of underlying signals. Based on a dataset containing groups of multiple partial observations of the same signal, as well as a differentiable model of the forward operator, this method can train a diffusion model for synthesizing underlying signals conditioned on a given partial observation. The underlying signals can then be projected into \"novel views\". They apply their method to inverse graphics, GAN inversion, and motion prediction. The method outperforms the tested baselines in several metrics.", "questions": "- How does your work compare to the many NeRF+Diffusion papers? Examples include [Q1, Q2] but there are many more. I ampositive that some of them apply to the same use case the authors present, and can be added as entries in Table 1.\n\n[Q1] https://arxiv.org/abs/2304.06714\n\n[Q2] https://arxiv.org/abs/2304.14473", "limitations": "No, the authors have not adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Results seem strong, both visually and quantitatively.\n- Paper is well-motivated, and addresses an important problem.\n- The metrics used to evaluate performance are sound and relevant.", "weaknesses": " - Proposition 1 is very vaguely written. It needs to be presented with more rigorous definitions (what are the assumptions in math form? what is the result? \"agrees with\" is not formal enough and the \"if\" statement is not clear to me).\n- More generally, the method section is presented in simplified terms. There needs to be more math there. Do you prove that your proposed loss function is indeed related to the maximum likelihood objective? KL divergence? What are the conditions (in math form) for Proposition 1? How does the sampling algorithm (after training) work? The overall presentation of the experiments section is also overly convoluted and unclear.\n- The requirement on the dataset to have multiple views of the same scene is still a significant assumption over the dataset. While this might make sense in 2D-to-3D, it makes less sense in other stochastic inverse problems such as accelerated MRI reconstruction (we almost never have two scans of the same object/person).\n\nMinor issues:\n- In lines 166-170, diffusion-based inverse problem solvers are mentioned. The main weakness of those works that is mentioned is their inability to learn from partial observations. It would be beneficial to discuss concurrent works [W1, W2] that model signals based on partial observations as well. What are the similarities and differences with your work?\n\n[W1] https://arxiv.org/abs/2305.13128\n\n[W2] https://arxiv.org/abs/2305.19256", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. The reviewer notes that our “results seem strong, both visually and quantitatively”, we “outperform the tested baselines”, and that our paper “addresses an important problem”. We now address the remaining concerns: \n\n### Proposition\nWe **intentionally** only provided a simplified version of the proposition in the main paper to preserve readability. **Note that we provide a rigorous statement and in-depth proof in the supplemental material** (Paper line 147: “...as we formally prove in the supplement:”). Following your comments, we will make the statement in the paper more rigorous and provide an abridged version of the proof already in the main paper. We will also make the references to the supplemental sections more prominent. \n\n### Loss Function\nIn the supplemental material, we **formally relate our loss function to the maximum likelihood objective**; see our supplemental Sec. 2 (“Our loss maximizes the likelihood over total observations”). Following your comment, we will make this explicitly clear in the main paper and reference the supplemental when discussing the loss. \n\n### Sampling at test time\nGiven a context observation, we start from pure noise and denoise the underlying signal and target observation, see L138-139 (“At test time, a signal is sampled by iterating Eq. 4, 5, and 6, ….”).  Also, see Figure 1 (bottom left “Single Denoising Step”). We iterate the single denoising step depicted here, starting from O^{trgt}_T. We will improve the exposition of the sampling stage and make it more prominent. For the inverse graphics application, refer to the updated overview figure; see general comment (G5).  \n\n### Presentation of the experiments\nWe provide details for each application in the supplemental document; see Section 3. We will work on making the exposition clearer and release our code to aid in understanding and reproduction. We have already updated the overview figure for the inverse graphics application; see general comment (G5).\n\n### Multiple views required for training\nWe require multiple observations during training; however, they do not necessarily need to cover the signal completely. Consider the 3D reconstruction application: During training, we only observe sparse camera views, observing only a small subset of each scene. Yet, our model can reconstruct complete scenes at test time, such as a complete 360 view of an object, by denoising the test camera trajectory. \n\nTo further demonstrate this, we conducted an experiment where we learn to reconstruct 64x64 res. images conditioned on partial observations that have a 32x32 patch missing. Referencing our formulation, the signal is the complete 2D image, and the forward model is a sampling function that can sample a specific region from the complete image. \nImportantly, we only use **incomplete** images during training, i.e., one patch from the image is always missing in our training dataset. From such training data, we first compute our context and target observations by applying further degradation to our training images - we remove pixels from the training image to compute the context, and the removed pixels become the target.\n\nOur method now learns to sample complete signals conditioned on the context observations. We use a similar architecture as our 3D reconstruction pipeline (see updated figure (G5)), where we condition the diffusion model on a deterministic estimate computed from the context. This network takes the context as input and computes a deterministic estimate of the signal we supervise only at the visible regions in the training images. The diffusion model is conditioned on this deterministic estimate and learns to sample the target patch by first reconstructing the signal, i.e., the complete image, and applying the forward model. At test time, we can complete the missing information in any image; see the response PDF. This shows that our method could likely apply to a wide range of stochastic inverse problems, including medical imaging. We also compare to the determinstic baseline in the response PDF. \n\n### Relation to concurrent works that learn from partial data\nAs correctly noted, they are concurrent to our submission and appear on arxiv after the submission deadline. The main difference is that both these papers seem to assume a linear forward model, while our approach supports a much wider range of forward models, as we demonstrate in our experiments. Our differentiable renderer is non-linear and GAN inversion also involves a non-linear forward pass of the GAN model. \n\n### Relation to NeRF+diffusion papers\nWe discuss many papers, including concurrent works, that use NeRF and diffusion, in Sec. 3 and Sec. 4.1.  The most closely related paper, in terms of the nature of the model, is RenderDiffusion, see general discussion comment (G1). The most closely related paper in terms of the task and the complexity of datasets is SparseFusio, which we extensively compare to and significantly outperform.\n\nWe briefly discuss the two papers: \nBoth [Q1] and [Q2] train 3D diffusion models using purely 3D architectures, adding noise to some 3D latents that represent the scene during training. As these 3D latents are not available when training from 2D observations, they either have to be pre-computed [Q2] or discovered jointly [Q1]. Our approach is radically different, where we show that learning to denoise observations can lead to sampling in 3D. We do not add noise to any unknown quantity at training time. Instead, we add noise to the known observation space. \n\nFurther, we show that our model scales to significantly more complex scenes, such as the compositional real indoor scenes from RealEstate10k, compared to the limited complexity of scenes considered in these two papers. \n\nPlease also note that both these papers are concurrent to our submission. \n\nThanks for pointing out all the references. We will cite them. \n\n### Limitations\nSee general discussion point (G4). "}]

Author rebuttal:
We thank all reviewers for their efforts. The reviewers note that we solve  “an essential problem in the present era” (5usH), we tackle a “key challenge” in training 3D diffusion models (bLEB), our architecture is “well-designed and intuitive” (LcVC), and “results seem strong, both visually and quantitatively” (zgt6). 

The reviewers have proposed additional experiments that will better highlight our work's strengths and limitations. **We are happy to report that we have been able to execute the majority of the experiments that the reviewers proposed, with favorable results, which we are excited to include in the paper.**. We now offer clarifications to some points shared across reviewers.  

### (G1) Differences to RenderDiffusion
RenderDiffusion indeed shares some of the motivation of our approach, which is training 3D diffusion models from 2D data, which we discuss in our paper (L162-163). However, our approach differs from RenderDiffusion in several critical points, which uniquely enables our method to go beyond simple, single-object scenes:
1. RenderDiffusion learns an unconditional model.
2. RenderDiffusion requires canonical camera poses, i.e., all of the objects have to be oriented according to a canonical reference frame. This is a major limitation, as canonical poses do not exist for compositional real-world scenes. Our approach does not require canonical poses. 
3. Through not requiring canonical camera poses, and with the proposed conditional diffusion that is not limited to monocular supervision, our model for the first time departs from simple, single-object scenes and succeeds at 3D generative modeling of room-scale, RealEstate10k scenes, a previously unachieved feat for 2D-to-3D generative models.

In addition, we provide a general theoretical formulation of conditional diffusion through forward models and define the conditions under which we can solve this problem optimally.  We also present empirical results for different applications. 

### (G2) Generality

bLEB appreciates the generality of our formulation but notes the lack of applications outside of vision. Our theoretical formulation is general and not limited to forward models in computer vision. We demonstrate results on distinct, diverse, and important vision problems. We already mention the scope of our experimental results in the paper (lines 13-14). However, we agree that further clarification is beneficial. We will update the introduction and abstract to clarify that we only demonstrate applications in computer vision and will mention other domains only in the future work section.

zgt6 mentions the requirement on multiple observations. While this is true, and we demonstrate several practical applications, we further explored the nature of observations necessary for training. We add an inpainting experiment using multiple observations created from partially observed signals during training. This task has parallels in many domains, such as medical imaging or audio processing, where the goal is to complete missing information. We provide more details for this experiment in the response to zgt6 (“Multiple views required for training”), and Tab.1, Fig.4 of the response PDF.  

More possible applications of our framework:
- Physics simulators can serve as forward models (see PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification [Li et al., 2023]) to learn the physical properties of objects by training on videos. 
- Physically-based sound synthesis as forward models (see Singing Voice Synthesis Using Differentiable LPC and Glottal-Flow-Inspired Wavetables [Yu and Fazekas, 2023]) can be used to predict parameters of human vocal chords by training on singing voice datasets.
 
We believe that our framework would enable uncertainty-aware inference of many more physical systems.  


### (G3) Evaluations


We evaluate against the state of the art in 3D reconstruction. While SparseFusion only uses the Co3D dataset for evaluation, we also use the very challenging scene-level  RealEstate10k dataset. We show results on more complex settings compared to the SparseFusion paper.  We only use a single input image (unlike >=2 for SparseFusion), and do not assume GT object masks, instead modeling the whole scene including background. This makes our task significantly more challenging. Thus, in our paper, we only trained on hydrants (a concurrent work, GeNVS, also only trained on hydrants). 

We agree with LcVC that training a general model on 10 Co3D categories will strengthen the paper. In the reponse PDF (Tab. 1, Fig. 2), we show the results of such a general model, and compare to pixelNeRF and SparseFusion, significantly outperforming them. Sparsefusion fails to generate any reasonable results in this multi-class setting. Note that the SparseFusion paper only trained category-specific models. **To the best of our knowledge, these are the first-ever results of category-agnostic diffusion models on Co3D.**
 
We further show preliminary results on training on renderings of 15k objects from the large-scale object-centric Objaverse dataset, see Fig. 3 of the response PDF. 

These results demonstrate that our method is capable of modeling complex distributions, and that it strongly outperforms all existing baselines. 

### (G4) Limitations
Reviewers asked about the limitations of our approach. They are mentioned in our supplemental pdf, see Section 4 (L191-202). Among other points, we note the complexity of training and inference. We will move the limitations to the main pdf in the revised version.

### (G5) Updated Figure
While most reviewers appreciate our exposition, zgt6 mentions that some application descriptions are unclear. We have updated the overview figure of our main application of inverse graphics that we hope better explains the main application, see Fig. 6 of the response PDF.


Meta-review: The reviewers were generally positive on the paper.  It proposes a novel approach to a challenging and interesting problem with solid results.  The paper is clearly above the bar and should be accepted.  I encourage the authors to further revise the manuscript based on the feedback from the review and discussion process for the final copy.
Decision: Accept (spotlight)